Opto-mechanics of panoramic capture devices with abutting cameras

ABSTRACT

A low parallax imaging device includes a plurality of imaging lens elements arranged to capture adjacent fields of view. In some examples, adjacent imaging lens elements may contact at datum features to maintain a desired spacing. The spacing may allow for partial overlapping of low-parallax volumes associated with the respective imaging lens elements.

CROSS REFERENCE TO RELATED APPLICATIONS

This disclosure claims benefit of priority of: U.S. Provisional PatentApplication Ser. No. 62/865,741, filed Jun. 24, 2019, entitled“Opto-Mechanics of Panoramic Capture Devices with Abutting Cameras;”U.S. Provisional Patent Application Ser. No. 62/952,973, filed Dec. 23,2019, entitled “Opto-Mechanics of Panoramic Capture Devices withAbutting Cameras;” U.S. Provisional Patent Application Ser. No.62/952,983, filed Dec. 23, 2019, entitled “Multi-camera Panoramic ImageCapture Devices with a Faceted Dome;” and U.S. Provisional PatentApplication Ser. No. 62/972,532, filed Feb. 10, 2020, entitled“Integrated Depth Sensing and Panoramic Camera System,” the entirety ofeach of which is incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to panoramic low-parallax multi-cameracapture devices having a plurality of adjacent and abutting polygonalcameras. The disclosure also relates the opto-mechanical design ofcameras that capture incident light from a polygonal shaped field ofview to form a polygonal shaped image.

BACKGROUND

Panoramic cameras have substantial value because of their ability tosimultaneously capture wide field of view images. The earliest suchexample is the fisheye lens, which is an ultra-wide-angle lens thatproduces strong visual distortion while capturing a wide panoramic orhemispherical image. While the field of view (FOV) of a fisheye lens isusually between 100 and 180 degrees, the approach has been extended toyet larger angles, including into the 220-270° range, as provided by Y.Shimizu in U.S. Pat. No. 3,524,697. As an alternative, there are mirroror reflective based cameras that capture annular panoramic images, suchas the system suggested by P. Greguss in U.S. Pat. No. 4,930,864. Whilethese technologies have continued to evolve, it is difficult for them toprovide a full hemispheric or spherical image with the resolution andimage quality that modern applications are now seeking.

As another alternative, panoramic multi-camera devices, with a pluralityof cameras arranged around a sphere or a circumference of a sphere, arebecoming increasingly common. However, in most of these systems,including those described in U.S. Pat. Nos. 9,451,162 and 9,911,454,both to A. Van Hoff et al., of Jaunt Inc., the plurality of cameras aresparsely populating the outer surface of the device. In order to capturecomplete 360-degree panoramic images, including for the gaps or seamsbetween the adjacent individual cameras, the cameras then have widenedFOVs that overlap one to another. In some cases, as much as 50% of acamera's FOV or resolution may be used for camera to camera overlap,which also creates substantial parallax differences between the capturedimages. Parallax is the visual perception that the position or directionof an object appears to be different when viewed from differentpositions. Then in the subsequent image processing, the excess imageoverlap and parallax differences both complicate and significantly slowthe efforts to properly combine, tile or stitch, and synthesizeacceptable images from the images captured by adjacent cameras.

There are also panoramic multi-camera devices in which a plurality ofcameras is arranged around a sphere or a circumference of a sphere, suchthat adjacent cameras are abutting along a part or the whole of adjacentedges. As an example, U.S. Pat. No. 7,515,177 by K. Yoshikawa depicts animaging device with a multitude of adjacent image pickup units(cameras). Images are collected from cameras having overlapping fieldsof view, so as to compensate for mechanical errors.

More broadly, in a multi-camera device, mechanical variations in theassembly and alignment of individual cameras, and of adjacent cameras toeach other, can cause real physical variations to both the camerasthemselves, and to the seam widths and parallelism of the camera edgesalong the seams. These variations can then affect the FOVs captured bythe individual cameras, the parallax errors in the images captured byadjacent cameras, the extent of “blind spots” in the FOV correspondingto the seams, the seam widths, and the amount of image overlap that isneeded to compensate. Thus, there are opportunities to improve panoramicmulti-camera devices and the low-parallax cameras thereof, relative tothe optical and opto-mechanical designs, and other aspects as well.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 depicts a 3D view of a portion of a multi-camera capture device,and specifically two adjacent cameras thereof.

FIGS. 2A and 2B depict portions of camera lens assemblies incross-section, including lens elements and ray paths.

FIG. 3 depicts a cross-sectional view of a portion of a standardmulti-camera capture device showing FOV overlap, Fields of view,overlap, seams, and blind regions.

FIG. 4 depicts two polyhedron shapes, that of a regular dodecahedron anda truncated icosahedron, to which a multi-camera capture device can bedesigned and fabricated.

FIG. 5A and FIG. 5B depict the optical geometry for fields of view foradjacent hexagonal and pentagonal lenses, as can occur with a devicehaving the geometry of a truncated icosahedron. FIG. 5B depicts anexpanded area of FIG. 5A with greater detail.

FIG. 5C depicts an example of a low parallax (LP) volume located nearboth a paraxial NP point or entrance pupil and a device center.

FIG. 5D depicts parallax differences for two adjacent cameras, relativeto a center of perspective.

FIG. 5E depicts front color at an edge of an outer compressor lenselement.

FIG. 6 depicts distortion correction curves plotted on a graph showing apercentage of distortion relative to a fractional field.

FIG. 7 depicts fields of view for adjacent cameras, including both Coreand Extended fields of view (FOV), both of which can be useful for thedesign of an optimized panoramic multi-camera capture device.

FIG. 8 depicts an image sensor with a sensor mount having adjustors.

FIG. 9 depicts a cross-sectional view of an improved opto-mechanicsconstruction for a multi-camera capture device, and a 3D view of acamera channel thereof.

FIG. 10 depicts a cross-sectional view of a seam and two adjacentstepped edge angle compressor lenses, near a seam, and a zoomed in viewof a portion (B) thereof.

FIG. 11 depicts 3D- and top-down views of a sensor area, masking, andoptical fiducial.

FIG. 12 depicts a baffle.

FIG. 13A depicts a cross-section of an improved lens design that showsmulti-compressor lens group and that can support a FRU concept.

FIG. 13B depicts another improved lens design, relative to an offsetdevice center.

FIG. 14A depicts a 3D view of a multi-camera capture device with fins.

FIG. 14B depicts a multi-camera capture device with pins.

FIG. 15 depicts an electronics system diagram for a multi-camera capturedevice.

FIG. 16 depicts different views of an outer compressor lens element withlens datums, and an associated lens housing with datum features.

FIG. 17 depicts a cross-sectional view of a portion of an assembly of animproved multi-camera panoramic image capture device, and a portionthereof in detail.

FIG. 18A depicts a series of views of components and features of a lenshousing and a channel loading support, that can be used during assemblyof an improved multi-camera panoramic image capture device.

FIG. 18B-1 depicts a cross-sectional views of an alternate version ofthe lens housings and the interaction thereof, at or near a seam, tothat shown in FIG. 10.

FIG. 18B-2 depicts a cross-sectional views of an alternate version ofthe channel to channel datums to those shown in FIG. 18A.

FIG. 19 depicts a view of an alternate design to that of FIG. 10 or FIG.17, for the mounting of the camera channels to a central support.

FIG. 20 depicts a view of another alternate design for the mounting ofthe camera channels to a central support.

FIG. 21 depicts an alternate configuration for an improved multi-cameraprojection device.

DETAILED DESCRIPTION

As is generally understood in the field of optics, a lens or lensassembly typically comprises a system or device having multiple lenselements which are mounted into a lens barrel or housing, and which worktogether to produce an optical image. An imaging lens captures a portionof the light coming from an object or plurality of objects that residein object space at some distance(s) from the lens system. The imaginglens can then form an image of these objects at an output “plane”; theimage having a finite size that depends on the magnification, asdetermined by the focal length of the imaging lens and the conjugatedistances to the object(s) and image plane, relative to that focallength. The amount of image light that transits the lens, from object toimage, depends in large part on the size of the aperture stop of theimaging lens, which is typically quantified by one or more values for anumerical aperture (NA) or an f-number (F# or F/#).

The image quality provided by the imaging lens depends on numerousproperties of the lens design, including the selection of opticalmaterials used in the design, the size, shapes (or curvatures) andthicknesses of the lens elements, the relative spacing of the lenselements one to another, the spectral bandwidth, polarization, lightload (power or flux) of the transiting light, optical diffraction orscattering, and/or lens manufacturing tolerances or errors. The imagequality is typically described or quantified in terms of lensaberrations (e.g., spherical, coma, or distortion), or the relative sizeof the resolvable spots provided by the lens, which is also oftenquantified by a modulation transfer function (MTF).

In a typical electronic or digital camera, an image sensor is nominallylocated at the image plane. This image sensor is typically a CCD or CMOSdevice, which is physically attached to a heat sink or other heatremoval means, and also includes electronics that provide power to thesensor, and read-out and communications circuitry that provide the imagedata to data storage or image processing electronics. The image sensortypically has a color filter array (CFA), such as a Bayer filter withinthe device, with the color filter pixels aligned in registration withthe image pixels to provide an array of RGB (Red, Green, Blue) pixels.Alternative filter array patterns, including the CYGM filter (cyan,yellow, green, magenta) or an RGBW filter array (W=white), can be usedinstead.

In typical use, many digital cameras are used by people or remotesystems in relative isolation, to capture images or pictures of a scene,without any dependence or interaction with any other camera devices. Insome cases, such as surveillance or security, the operation of a cameramay be directed by people or algorithms based on image content seen fromanother camera that has already captured overlapping, adjacent, orproximate image content. In another example, people capture panoramicimages of a scene with an extended or wide FOV, such as a landscapescene, by sequentially capturing a sequence of adjacent images, whilemanually or automatically moving or pivoting to frame the adjacentimages. Afterwards, image processing software, such as Photoshop orLightroom, can be used to stitch, mosaic, or tile the adjacent imagestogether to portray the larger extended scene. Image stitching or photostitching is the process of combining multiple photographic images withoverlapping fields of view to produce a segmented panorama orhigh-resolution image. Image quality improvements, including exposure orcolor corrections, can also be applied, either in real time, or in apost processing or image rendering phase, or a combination thereof.

Unless the objects in a scene are directionally illuminated and/or havea directional optical response (e.g., such as with reflectance), theavailable light is plenoptic, meaning that there is light travelling inevery direction, or nearly so, in a given space or environment. A cameracan then sample a subset of this light, as image light, with which itprovides a resulting image that shows a given view or perspective of thedifferent objects in the scene at one or more instants in time. If thecamera is moved to a different nearby location and used to captureanother image of part of that same scene, both the apparent perspectivesand relative positioning of the objects will change. In the latter case,one object may now partially occlude another, while a previously hiddenobject becomes at least partially visible. These differences in theapparent position or direction of an object are known as parallax. Inparticular, parallax is a displacement or difference in the apparentposition of an object viewed along two different lines of sight and ismeasured by the angle or semi-angle of inclination between those twolines.

In a stereoscopic image capture or projection system, dual view parallaxis a cue, along with shadowing, occlusion, and perspective, that canprovide a sense of depth. For example, in a stereo (3D) projectionsystem, polarization or spectrally encoded image pairs can be overlapprojected onto a screen to be viewed by audience members wearingappropriate glasses. The amount of parallax can have an optimal range,outside of which, the resulting sense of depth can be too small toreally be noticed by the audience members, or too large to properly befused by the human visual system.

Whereas, in a panoramic image capture application, parallax differencescan be regarded as an error that can complicate both image stitching andappearance. In the example of an individual manually capturing apanoramic sequence of landscape images, the visual differences inperspective or parallax across images may be too small to notice if theobjects in the scene are sufficiently distant (e.g., optically atinfinity). An integrated panoramic capture device with a rotating cameraor multiple cameras has the potential to continuously capture real timeimage data at high resolution without being dependent on theuncertainties of manual capture. But such a device can also introduceits own visual disparities, image artifacts, or errors, including thoseof parallax, perspective, and exposure. Although the resulting imagescan often be successfully stitched together with image processingalgorithms, the input image errors complicate and lengthen imageprocessing time, while sometimes leaving visually obvious residualerrors.

To provide context, FIG. 1 depicts a portion of an improved integratedpanoramic multi-camera capture device 100 having two adjacent cameras120 in housings 130 which are designed for reduced parallax imagecapture. These cameras are alternately referred to as image pick-upunits, or camera channels, or objective lens systems. The cameras 120each have a plurality of lens elements (see FIG. 2) that are mountedwithin a lens barrel or housing 130. The adjacent outer lens elements137 have adjacent beveled edges 132 and are proximately located, onecamera channel to another, but which may not be in contact, and thus areseparated by a gap or seam 160 of finite width. Some portion of theavailable light (□), or light rays 110, from a scene or object space 105will enter a camera 120 to become image light that was captured within aconstrained FOV and directed to an image plane, while other light rayswill miss the cameras entirely. Some light rays 110 will propagate intothe camera and transit the constituent lens elements as edge-of-fieldchief rays 170, or perimeter rays, while other light rays canpotentially propagate through the lens elements to create stray or ghostlight and erroneous bright spots or images. As an example, some lightrays (167) that are incident at large angles to the outer surface of anouter lens element 137 can transit a complex path through the lenselements of a camera and create a detectable ghost image at the imageplane 150.

In greater detail, FIG. 2A depicts a cross-section of part of a camera120 having a set of lens elements 135 mounted in a housing (130, notshown) within a portion of an integrated panoramic multi-camera capturedevice 100. A fan of light rays 110 from object space 105, spanning therange from on axis to full field off axis chief rays, are incident ontothe outer lens element 137, and are refracted and transmitted inwards.This image light 115 that is refracted and transmitted through furtherinner lens elements 140, through an aperture stop 145, converges to afocused image at or near an image plane 150, where an image sensor (notshown) is typically located. The lens system 120 of FIG. 2A can also bedefined as having a lens form that consists of outer lens element 137 orcompressor lens element, and inner lens elements 140, the latter ofwhich can also be defined as consisting of a pre-stop wide angle lensgroup, and a post-stop eyepiece-like lens group. This compressor lenselement (137) directs the image light 115 sharply inwards, compressingthe light, to both help enable the overall lens assembly to provide ashort focal length, while also enabling the needed room for the cameralens housing or barrel to provide the mechanical features necessary toboth hold or mount the lens elements and to interface properly with thebarrel or housing of an adjacent camera. The image light that transiteda camera lens assembly from the outer lens element 137 to the imageplane 150 will provide an image having an image quality, that can bequantified by an image resolution, image contrast, a depth of focus, andother attributes, whose quality was defined by the optical aberrations(e.g., astigmatism, distortion, or spherical) and chromatic or spectralaberrations, encountered by the transiting light at each of the lenselements (137, 140) within a camera 120. FIG. 2B depicts a fan of chiefrays 170, or perimeter rays, incident along or near a beveled edge 132of the outer lens element 137 of the camera optics (120) depicted inFIG. 2A. FIG. 2B also depicts a portion of a captured, polygonal shapedor asymmetrical, FOV 125, that extends from the optical axis 185 to aline coincident with an edge ray.

In the camera lens design depicted in FIG. 2A, the outer lens element137 functions as a compressor lens element that redirects the transitingimage light 115 towards a second lens element 142, which is the firstlens element of the group of inner lens elements 140. In this design,this second lens element 142 has a very concave shape that isreminiscent of the outer lens element used in a fish-eye type imaginglens. This compressor lens element directs the image light 115 sharplyinwards, or bends the light rays, to both help enable the overall lensassembly to provide a short focal length, while also enabling the neededroom for the camera lens housing 130 or barrel to provide the mechanicalfeatures necessary to both hold or mount the lens elements 135 and tointerface properly with the barrel or housing of an adjacent camera.However, with a good lens and opto-mechanical design, and an appropriatesensor choice, a camera 120 can be designed with a lens assembly thatsupports an image resolution of 20-30 pixels/degree, to as much as 110pixels/degree, or greater, depending on the application and the deviceconfiguration.

The resultant image quality from these cameras will also depend on thelight that scatters at surfaces, or within the lens elements, and on thelight that is reflected or transmitted at each lens surface. The surfacetransmittance and camera lens system efficiency can be improved by theuse of anti-reflection (AR) coatings. The image quality can also dependon the outcomes of non-image light. Considering again FIG. 1, otherportions of the available light can be predominately reflected off ofthe outer lens element 137. Yet other light that enters a camera 120 canbe blocked or absorbed by some combination of blackened areas (notshown) that are provided at or near the aperture stop, the inner lensbarrel surfaces, the lens element edges, internal baffles or lighttrapping features, a field stop, or other surfaces. Yet other light thatenters a camera can become stray light or ghost light 167 that is alsopotentially visible at the image plane.

The aggregate image quality obtained by a plurality of adjacent cameras120 within an improved integrated panoramic multi-camera capture device100 (e.g., FIG. 1) can also depend upon a variety of other factorsincluding the camera to camera variations in the focal length and/ortrack length, and magnification, provided by the individual cameras.These parameters can vary depending on factors including the variationsof the glass refractive indices, variations in lens element thicknessesand curvatures, and variations in lens element mounting. As an example,images that are tiled or mosaiced together from a plurality of adjacentcameras will typically need to be corrected, one to the other, tocompensate for image size variations that originate with cameramagnification differences (e.g., ±2%).

The images produced by a plurality of cameras in an integrated panoramicmulti-camera capture device 100 can also vary in other ways that effectimage quality and image mosaicing or tiling. In particular, thedirectional pointing or collection of image light through the lenselements to the image sensor of any given camera 120 can vary, such thatthe camera captures an angularly skewed or asymmetrical FOV (FOV ormis-sized FOV (FOV±). The lens pointing variations can occur duringfabrication of the camera (e.g., lens elements, sensor, and housing) orduring the combined assembly of the multiple cameras into an integratedpanoramic multi-camera capture device 100, such that the alignment ofthe individual cameras is skewed by misalignments or mounting stresses.When these camera pointing errors are combined with the presence of theseams 160 between cameras 120, images for portions of an availablelandscape or panoramic FOV that may be captured, may instead be missedor captured improperly. The variabilities of the camera pointing, andseams can be exacerbated by mechanical shifts and distortions that arecaused by internal or external environmental factors, such as heat orlight (e.g., image content), and particularly asymmetrical loadsthereof.

In comparison to the FIG. 1 system, in a typical commercially availablepanoramic camera, the seams between cameras are outright gaps that canbe 30-50 mm wide, or more. In particular, as shown in FIG. 3, apanoramic multi-camera capture device 101 can have adjacent cameras 120or camera channels separated by large gaps or seams 160, between whichthere are blind spots or regions 165 from which neither camera cancapture images. The actual physical seams 160 between adjacent camerachannels or outer lens elements 137 (FIG. 1 and FIG. 3) can be measuredin various ways; as an actual physical distance between adjacent lenselements or lens housings, as an angular extent of lost FOV, or as anumber of “lost” pixels. However, the optical seam, as the distancebetween outer chief rays of one camera to another can be larger yet, dueto any gaps in light acceptance caused by vignetting or coating limits.For example, anti-reflection (AR) coatings are not typically depositedto the edges of optics, but an offsetting margin is provided, to providea coated clear aperture (CA).

To compensate for both camera misalignments and the large seams 160, andto reduce the size of the blind regions 165, the typical panoramicmulti-camera capture devices 101 (FIG. 3) have each of the individualcameras 120 capture image light 115 from wide FOVs 125 that provideoverlap 127, so that blind regions 165 are reduced, and the potentialcapturable image content that is lost is small. As another example, inmost of the commercially available multi-camera capture devices 101, thegaps are 25-50+ mm wide, and the compensating FOV overlap betweencameras is likewise large; e.g., the portions of the FOVs 125 that areoverlapping and are captured by two adjacent cameras 120 can be as muchas 10-50% of a camera's FOV. The presence of such large image overlapsfrom shared FOVs 125 wastes potential image resolution and increases theimage processing and image stitching time, while introducing significantimage parallax and perspective errors. These errors complicate imagestitching, as the errors must be corrected or averaged during thestitching process. In such systems, the parallax is not predictablebecause it changes as a function of object distance. If the objectdistance is known, the parallax can be predicted for given fields ofview and spacing between cameras. But because the object distance is nottypically known, parallax errors then complicate image stitching.Optical flow and common stitching algorithms determine an object depthand enable image stitching, but with processing power and time burdens.

Similarly, in a panoramic multi-camera capture device 100, of the typeof FIG. 1, with closely integrated cameras, the width and constructionat the seams 160 can be an important factor in the operation of theentire device. However, the seams can be made smaller than in FIG. 3,with the effective optical seam width between the FOV edges of twoadjacent cameras determined by both optical and mechanicalcontributions. For example, by using standard optical engineeringpractices to build lens assemblies in housings, the mechanical width ofthe seams 160 between the outer lens elements 137 of adjacent camerasmight be reduced to 4-6 mm. For example, it is standard practice toassemble lens elements into a lens barrel or housing that has a minimumradial width of 1-1.5 mm, particularly near the outermost lens element.Then accounting for standard coated clear apertures or coating margins,and accounting for possible vignetting, aberrations of the entrancepupil, front color, chip edges, and trying to mount adjacent lensassemblies or housings in proximity by standard techniques. Thus, whenaccounting for both optics and mechanics, an optical seam width betweenadjacent lenses can easily be 8-12 mm or more.

But improved versions of the panoramic multi-camera capture device (300)of the type of FIG. 1, with optical and opto-mechanical designs thatenable significantly smaller seams, and with further improved parallaxperformance, are possible. As a first example, for the presenttechnology for improved polygonal shaped cameras, during early stages offabrication of outer lens elements 137, these lenses can have a circularshape and can be AR coated to at or near their physical edges. Whenthese lenses are subsequently processed to add the polygonal shapedefining beveled edges 132 (e.g., FIG. 2B), a result can be that the ARcoatings will essentially extend to the beveled lens edges. Theeffective optical or coated clear apertures can then defined by anyallowances for mechanical mounting or for the standard edge grind thatis used in optics manufacturing to avoid edge chipping. With thisapproach, and a mix of other techniques that will be subsequentlydiscussed, the optical seams can be reduced to 1-5 mm width.

Aspects of the present disclosure produce high quality low-parallaxpanoramic images from an improved multi-camera panoramic capture device(300), for which portions of a first example are shown in FIG. 8 andFIG. 9. This broad goal can be enabled by developing a systemic range ofdesign strategies to inform both the optical and opto-mechanical lensdesign efforts, and the opto-mechanical device design and fabricationefforts, as well as strategies for improved image capture andprocessing. This goal can also be enabled by providing for both initialand ongoing camera and device calibration. In broad terms, the imageprocessing or rendering of images is a method to generate quality imagesfrom the raw captured image data that depends on the camera intrinsics(geometric factors such as focal length and distortion), the cameraextrinsics (geometric factors such as camera orientation to objectspace), other camera parameters such as vignetting and transmission, andillumination parameters such as color and directionality. With respectto an improved multi-camera panoramic capture device 300, the use offiducials in determining and tracking a center pixel or an imagecentroid, exposure correction, and knowledge of the camera intrinsicsfor any given camera 320 in a device, are all assists towards completingreliable and repeatable tiling of images obtained from a plurality ofadjacent cameras. Thus the subsequent discussions are broadly focused onproviding optical (camera or objective lens) designs that can enable thedesired image quality, as well as camera and device assembly approaches,management of key tolerances, camera calibration, knowledge of cameraintrisincs and extriniscs, and other factors that can likewise affectthe resultant device performance. The improved panoramic multi-cameracapture devices of the present invention can be used to support a widevariety of applications or markets, including cinematic image capture,augmented reality or virtual reality (VR) image capture, surveillance orsecurity imaging, sports or event imaging, mapping or photogrammetry,vehicular navigation, and robotics.

Before exploring opto-mechanical means for enabling improved panoramicmulti-camera capture devices (300), means for providing cameras 120 thatare improved for use in these systems are developed. Accordingly, thegoals include providing improved cameras (320) having both reducedparallax errors and image overlap. As one aspect of the presentapproach, a goal is to reduce the residual parallax error for the edgechief rays collected respectively by each camera in an adjacent pair.The parallax error is defined as the change in parallax with respect toobject distance (e.g., that the chief ray trajectory with respect to anear distance (e.g., 3 feet) from the device, versus a far distance(e.g., 1 mile), is slightly different). For example, as one goal ortarget for reduced parallax, or to have effectively no parallax error,or to be “parallax-free”, is that the chief rays of adjacent camerasshould deviate from parallelism to each other by ≤0.5-2.0 deg., andpreferably by ≤0.01-0.1 deg. Alternately, or equivalently, the parallaxerror, as assessed as a perspective error in terms of location on theimage plane, should be reduced to ≤2 pixels, and preferably to ≤0.5pixel. As another aspect of the present approach, the width of the seams160 between adjacent cameras (e.g., 120, 320) assembled into their ownlens housings are to be reduced. The goal is to reduce the width of theseams, both in terms of their absolute physical width, and their opticalwidth or an effective width. For example, a goal is to reduce a seam 160between adjacent outer lens elements 137 to having a maximum gap or anactual physical seam width in a range of only 0.5-3.0 mm, and to thenreduce the maximum optical seam width to a range of about only 1-6 mm.As an example, these reduced seams widths can translate to a reducedangular extent of lost FOV of only 0.25-1.0°, or a number of “lost”pixels of only 2-20 pixels. For example, for a device providing 8kpixels around a 360-degree panorama equirectangular image, a loss ofonly 2-4 pixels at the seams can be acceptable as the residual imageartifacts can be difficult to perceive. The actual details or numericaltargets for effectively no-parallax error, or for the maximum opticalseam width, depend on many factors including the detailedopto-mechanical designs of the improved cameras (320) and overall device(300), management of tolerances, possible allowances for a center offsetdistance or an amount of extended FOV (215) and the targets for lowparallax therein, and the overall device specifications (e.g., diameter,sensor resolution or used sensor pixels within an imaged FOV or a CoreFOV 205 (FIG. 7)). Further goals, enabled by some combination of theabove improvements, are for each camera to reliably and quickly provideoutput images from an embedded sensor package that are cropped down toprovide core FOV images, and then that each cropped image can be readilyseamed or tiled with cropped images provided by adjacent cameras, so asto readily provide panoramic output images from an improved multi-cameracapture device (300) in real time.

An improved panoramic multi-camera capture device 300, such as that ofFIG. 13 and FIG. 15, can have a plurality of cameras arranged around acircumference of a sphere to capture a 360-degree annular FOV.Alternately, a panoramic multi-camera capture device can have aplurality of cameras arranged around a spherical or polyhedral shape. Apolyhedron is a three-dimensional solid consisting of a collection ofpolygons that are contiguous at the edges. One polyhedral shape, asshown in FIG. 4, is that of a dodecahedron 50, which has 12 sides orfaces, each shaped as a regular pentagon 55, and 20 vertices or corners(e.g., a vertex 60). A panoramic multi-camera capture device formed tothe dodecahedron shape has cameras with a pentagonally shaped outer lenselements that nominally image a 69.1° full width field of view. Anothershape is that of a truncated icosahedron, like a soccer ball, which asis also shown in FIG. 4, and has a combination of 12 regular pentagonalsides or faces, 20 regular hexagonal sides or faces, 60 vertices, and 90edges. More complex shapes, with many more sides, such as regularpolyhedra, Goldberg polyhedra, or shapes with octagonal sides, or evensome irregular polyhedral shapes, can also be useful. For example, aGoldberg chamfered dodecahedron is similar to the truncated icosahedron,with both pentagonal and hexagonal facets, totaling 42 sides. But ingeneral, the preferred polyhedrons for the current purpose have sides orfaces that are hexagonal or pentagonal, which are generally roundishshapes with beveled edges 132 meeting at obtuse corners. Otherpolyhedral shapes, such as an octahedron or a regular icosahedron can beused, although they have triangular facets. Polyhedral facets with moreabrupt or acute corners, such as square or triangular faces, can beeasier to fabricate, as compared to facets with pentagonal and orhexagonal facets, as they have fewer edges to cut to provide polygonaledges on the outermost lens element, so as to define a capturedpolygonal FOV. However, greater care can then be needed in cutting,beveling, and handling the optic because of those acute corners.Additionally, for lens facets with large FOVs and acute facet angles, itcan be more difficult to design the camera lenses and camera lenshousings for optical and opto-mechanical performance. Typically, a 360°polyhedral camera will not capture a full spherical FOV as at least partof one facet is sacrificed to allow for support features and power andcommunications cabling, such as via a mounting post. However, if thedevice communicates wirelessly, and is also hung by a thin cable to avertex, the FOV lost to such physical connections can be reduced.

As depicted in FIG. 1 and FIG. 2B, a camera channel 120 can resembles afrustum, or a portion thereof, where a frustum is a geometric solid(normally a cone or pyramid) that lies between one or two parallelplanes that cut through it. In that context, a fan of chief rays 170corresponding to a polygonal edge, can be refracted by an outercompressor lens element 137 to nominally match the frustum edges inpolyhedral geometries.

To help illustrate some issues relating to camera geometry, FIG. 5Aillustrates a cross-sections of a pentagonal lens 175 capturing apentagonal FOV 177 and a hexagonal lens 180 capturing a hexagonal FOV182, representing a pair of adjacent cameras whose outer lens elementshave pentagonal and hexagonal shapes, as can occur with a truncatedicosahedron, or soccer ball type panoramic multi-camera capture devices(e.g., 100, 300). The theoretical hexagonal FOV 182 spans a half FOV of20.9°, or a full FOV of 41.8° (□₁) along the sides, although the FOVnear the vertices is larger. The pentagonal FOV 177 supports 36.55° FOV(□₂) within a circular region, and larger FOVs near the corners orvertices. Notably, in this cross-section, the pentagonal FOV 177 isasymmetrical, supporting a 20-degree FOV on one side of an optical axis185, and only a 16.5-degree FOV on the other side of the optical axis.

Optical lenses are typically designed using programs such as ZEMAX orCode V. Design success typically depends, in part, on selecting the bestor most appropriate lens parameters, identified as operands, to use inthe merit function. This is also true when designing a lens system foran improved low-parallax multi-camera panoramic capture device (300),for which there are several factors that affect performance (including,particularly parallax) and several parameters that can be individuallyor collectively optimized, so as to control it. One approach targetsoptimization of the “NP” point, or more significantly, variants thereof.

As background, in the field of optics, there is a concept of theentrance pupil, which is a projected image of the aperture stop as seenfrom object space, or a virtual aperture which the imaged light raysfrom object space appear to propagate towards before any refraction bythe first lens element. By standard practice, the location of theentrance pupil can be found by identifying a paraxial chief ray fromobject space 105, that transits through the center of the aperture stop,and projecting or extending its object space direction forward to thelocation where it hits the optical axis 185. In optics, incident Gaussor paraxial rays are understood to reside within an angular range ≤10°from the optical axis, and correspond to rays that are directed towardsthe center of the aperture stop, and which also define the entrancepupil position. Depending on the lens properties, the entrance pupil maybe bigger or smaller than the aperture stop, and located in front of, orbehind, the aperture stop.

By comparison, in the field of low-parallax cameras, there is a conceptof a no-parallax (NP) point, or viewpoint center. Conceptually, the “NPPoint” has been associated with a high FOV chief ray or principal rayincident at or near the outer edge of the outermost lens element, andprojecting or extending its object space direction forward to thelocation where it hits the optical axis 185. For example, depending onthe design, camera channels in a panoramic multi-camera capture devicecan support half FOVs with non-paraxial chief rays at angles >31° for adodecahedron type system (FIG. 4) or >20° for a truncated icosahedrontype system (see FIG. 4 and FIG. 5A). This concept of the NP pointprojection has been applied to the design of panoramic multi-cameracapture devices, relative to the expectations for chief ray propagationand parallax control for adjacent optical systems (cameras). It is alsostated that if a camera is pivoted about the NP point, or a plurality ofcamera's appear to rotate about a common NP point, then parallax errorswill be reduced, and images can be aligned with little or no parallaxerror or perspective differences. But in the field of low parallaxcameras, the NP point has also been equated to the entrance pupil, andthe axial location of the entrance pupil that is estimated using a firstorder optics tangent relationship between a projection of a paraxialfield angle and the incident ray height at the first lens element (seeFIGS. 2A, 2B).

Thus, confusingly, in the field of designing of low-parallax cameras,the NP point has also been previously associated with both with theprojection of edge of FOV chief rays and the projection of chief raysthat are within the Gauss or paraxial regime. As will be seen, inactuality, they both have value. In particular, an NP point associatedwith the paraxial entrance pupil can be helpful in developing initialspecifications for designing the lens, and for describing the lens. AnNP point associated with non-paraxial edge of field chief rays can beuseful in targeting and understanding parallax performance and indefining the conical volume or frustum that the lens assembly can residein.

The projection of these non-paraxial chief rays can miss the paraxialchief ray defined entrance pupil because of both lens aberrations andpractical geometry related factors associated with these lens systems.Relative to the former, in a well-designed lens, image quality at animage plane is typically prioritized by limiting the impact ofaberrations on resolution, telecentricity, and other attributes. Withina lens system, aberrations at interim surfaces, including the aperturestop, can vary widely, as the emphasis is on the net sums at the imageplane. Aberrations at the aperture stop are often somewhat controlled toavoid vignetting, but a non-paraxial chief ray need not transit thecenter of the aperture stop or the projected paraxially located entrancepupil.

To expand on these concepts, and to enable the design of improved lowparallax lens systems, it is noted that the camera lens system 120 inFIG. 2A depicts both a first NP point 190A, corresponding to theentrance pupil as defined by a vectoral projection of paraxial chiefrays from object space 105, and an offset second NP point 190B,corresponding to a vectoral projection of a non-paraxial chief rays fromobject space. Both of these ray projections cross the optical axis 185in locations behind both the lens system and the image plane 150. Aswill be subsequently discussed, the ray behavior in the region betweenand proximate to the projected points 190A and 190B can be complicatedand neither projected location or point has a definitive value or size.A projection of a chief ray will cross the optical axis at a point, buta projection of a group of chief rays will converge towards the opticalaxis and cross at different locations, that can be tightly clustered(e.g., within a few or tens of microns), where the extent or size ofthat “point” can depends on the collection of proximate chief rays usedin the analysis. Whereas, when designing low parallax imaging lensesthat image large FOVs, the axial distance or difference between the NPpoints 190A and 190B that are provided by the projected paraxial andnon-paraxial chief rays can be significantly larger (e.g., millimeters).Thus, as will also be discussed, the axial difference represents avaluable measure of the parallax optimization (e.g., a low parallaxvolume 188) of a lens system designed for the current panoramic capturedevices and applications. As will also be seen, the design of animproved device (300) can be optimized to position the geometric centerof the device, or device center 196, outside, but proximate to this lowparallax volume 188, or alternately within it, and preferably proximateto a non-paraxial chief ray NP point.

As one aspect, FIG. 5A depicts the projection of the theoretical edge ofthe fields of view (FOV edges 155), past the outer lens elements (lenses175 and 180) of two adjacent cameras, to provide lines directed to acommon point (190). These lines represent theoretical limits of thecomplex “conical” opto-mechanical lens assemblies, which typically arepentagonally conical or hexagonally conical limiting volumes. Again,ideally, in a no-parallax multi-camera system, the entrance pupils or NPpoints of two adjacent cameras are co-located. But to avoid mechanicalconflicts, the mechanics of a given lens assembly, including the sensorpackage, should generally not protrude outside a frustum of a camerasystem and into the conical space of an adjacent lens assembly. However,real lens assemblies in a multi-camera panoramic capture device are alsoseparated by seams 160. Thus, the real chief rays 170 that are acceptedat the lens edges, which are inside of both the mechanical seams and aphysical width or clear aperture of a mounted outer lens element (lenses175 and 180), when projected generally towards a paraxial NP point 190,can land instead at offset NP points 192, and be separated by an NPpoint offset distance 194.

This can be better understood by considering the expanded area A-A inproximity to a nominal or ideal point NP 190, as shown in detail in FIG.5B. Within a hexagonal FOV 182, light rays that propagate within theGauss or paraxial region (e.g., paraxial ray 173), and that pass throughthe nominal center of the aperture stop, can be projected to a nominalNP point 190 (corresponding to the entrance pupil), or to an offset NPpoint 190A at a small NP point difference or offset 193 from a nominalNP point 190. Whereas, the real hexagonal lens edge chief rays 170associated with a maximum inscribed circle within a hexagon, can projectto land at a common offset NP point 192A that can be at a larger offsetdistance (194A). The two adjacent cameras in FIGS. 5A,B also may or maynot share coincident NP points (e.g., 190). Distance offsets can occurdue to various reasons, including geometrical concerns between cameras(adjacent hexagonal and pentagonal cameras), geometrical asymmetrieswithin a camera (e.g., for a pentagonal camera), or from limitationsfrom the practical widths of seams 160, or because of the directionalitydifference amongst aberrated rays.

As just noted, there are also potential geometric differences in theprojection of incident chief rays towards a simplistic nominal “NPpoint” (190). First, incident imaging light paths from near the cornersor vertices or mid-edges (mid-chords) of the hexagonal or pentagonallenses may or may not project to common NP points within the describedrange between the nominal paraxial NP point 190 and an offset NP point192B. Also, as shown in FIG. 5B, just from the geometric asymmetry ofthe pentagonal lenses, the associated pair of edge chief rays 170 and171 for the real accepted FOV, can project to different nominal NPpoints 192B that can be separated from both a paraxial NP point (190) byan offset distance 194B and from each other by an offset distance 194C.

As another issue, during lens design, the best performance typicallyoccurs on axis, or near on axis (e.g., ≤0.3 field (normalized)), nearthe optical axis 185. In many lenses, good imaging performance, bydesign, often occurs at or near the field edges, where optimizationweighting is often used to force compliance. The worst imagingperformance can then occur at intermediate fields (e.g., 0.7-0.8 of anormalized image field height). Considering again FIG. 5A,B,intermediate off axis rays, from intermediate fields (θ) outside theparaxial region, but not as extreme as the edge chief rays(10°<θ<20.9°), can project towards intermediate NP points between anominal NP point 190 and an offset NP point 192B. But other, moreextreme off axis rays, particularly from the 0.7-0.8 intermediatefields, that are more affected by aberrations, can project to NP pointsat locations that are more or less offset from the nominal NP point 190than are the edge of field offset NP points 192B. Accounting for thevariations in lens design, the non-paraxial offset “NP” points can falleither before (closer to the lens) the paraxial NP point (the entrancepupil) as suggested in FIG. 5B, or after it (as shown in FIG. 2A).

This is shown in greater detail in FIG. 5C, which essentiallyillustrates a further zoomed-in region A-A of FIG. 5B, but whichillustrates an impact from vectoral projected ray paths associated withaberrated image rays, that converge at and near the paraxial entrancepupil (190), for an imaging lens system that was designed and optimizedusing the methods of the present approach. In FIG. 5C, the projected raypaths of green aberrated image rays at multiple fields from a cameralens system converge within a low parallax volume 188 near one or more“NP” points. Similar illustrations of ray fans can also be generated forRed or Blue light. The projection of paraxial rays 173 can converge ator near a nominal paraxial NP point 190, or entrance pupil, located on anominal optical axis 185 at a distance Z behind the image plane 150. Theprojection of edge of field rays 172, including chief rays 171, convergeat or near an offset NP point 192B along the optical axis 185. The NPpoint 192B can be quantitatively defined, for example, as the center ofmass of all edge of field rays 172. An alternate offset NP point 192Acan be identified, that corresponds to a “circle of least confusion”,where the paraxial, edge, and intermediate or mid-field rays, aggregateto the smallest spot. These different “NP” points are separated from theparaxial NP point by offset distances 194A and 194B, and from each otherby an offset distance 194C. Thus, it can be understood that an aggregate“NP point” for any given real imaging lens assembly or camera lens thatsupports a larger than paraxial FOV, or an asymmetrical FOV, istypically not a point, but instead can be an offset low parallax (LP)smudge or volume 188.

Within a smudge or low parallax volume 188, a variety of possibleoptimal or preferred NP points can be identified. For example, an offsetNP point corresponding to the edge of field rays 172 can be emphasized,so as to help provide improved image tiling. An alternate mid-field(e.g., 0.6-0.8) NP point (not shown) can also be tracked and optimizedfor. Also the size and position of the overall “LP” smudge or volume188, or a preferred NP point (e.g., 192B) therein, can change dependingon the lens design optimization. Such parameters can also vary amongstlenses, for one fabricated lens system of a given design to another, dueto manufacturing differences amongst lens assemblies. Although FIG. 5Cdepicts these alternate offset “NP points” 192A,B for non-paraxial raysas being located after the paraxial NP point 190, or further away fromthe lens and image plane, other lenses of this type, optimized using themethods of the present approach, can be provided where similarnon-paraxial NP points 192A,B that are located with a low parallaxvolume 188 can occur at positions between the image plane and theparaxial NP point.

FIG. 5C also shows a location for a center of the low-parallaxmulti-camera panoramic capture device, device center 196. Based onoptical considerations, an improved panoramic multi-camera capturedevice 300 can be preferably optimized to nominally position the devicecenter 196 within the low parallax volume 188. Optimized locationstherein can include being located at or proximate either of the offsetNP points 192A or 192B, or within the offset distance 194B between them,so as to prioritize parallax control for the edge of field chief rays.The actual position therein depends on parallax optimization, which canbe determined by the lens optimization relative to spherical aberrationof the entrance pupil, or direct chief ray constraints, or distortion,or a combination thereof. For example, whether the spherical aberrationis optimized to be over corrected or under corrected, and how weightingson the field operands in the merit function are used, can affect thepositioning of non-paraxial “NP” points for peripheral fields or midfields. The “NP” point positioning can also depend on the management offabrication tolerances and the residual variations in lens systemfabrication. The device center 196 can also be located proximate to, butoffset from the low parallax volume 188, by a center offset distance198. This approach can also help tolerance management and provide morespace near the device center 196 for cables, circuitry, coolinghardware, and the associated structures. In such case, the adjacentcameras 120 can then have offset low parallax volumes 188 of “NP” points(FIG. 5D), instead of coincident ones (FIGS. 5A, B). In this example, ifthe device center 196 is instead located at or proximate to the paraxialentrance pupil, NP point 190, then effectively one or more of the outerlens elements 137 of the cameras 120 are undersized and the desired fullFOVs are not achievable. FIG. 13B depicts the possible positioning of asimilar lens system 920 with respect to an offset device center 910.

Thus, while the no-parallax (NP) point is a useful concept to worktowards, and which can valuably inform panoramic image capture andsystems design, and aid the design of low-parallax error lenses, it isidealized, and its limitations must also be understood. Considering thisdiscussion of the NP point(s) and LP smudges, in enabling an improvedlow-parallax multi-camera panoramic capture device (lens design exampleto follow; device 300 of FIG. 13A), it is important to understand raybehavior in this regime, and to define appropriate parameters oroperands to optimize, and appropriate target levels of performance toaim for. In the latter case, for example, a low parallax lens with atrack length of 65-70 mm can be designed for in which the LP smudge isas much as 10 mm wide (e.g., offset distance 194A). But alternate lensdesigns, for which this parameter is further improved, can have a lowparallax volume 188 with a longitudinal LP smudge width or width alongthe optical axis (offset 194A) of a few millimeters or less.

The width and location of the low parallax volume 188, and the vectoraldirections of the projections of the various chief rays, and their NPpoint locations within a low parallax volume, can be controlled duringlens optimization by a method using operands associated with a fan ofchief rays 170 (e.g., FIGS. 2A,B). But the LP smudge or LP volume 188 ofFIG. 5C can also be understood as being a visualization of thetransverse component of spherical aberration of the entrance pupil, andthis parameter can be used in an alternate, but equivalent, designoptimization method to using chief ray fans. In particular, during lensoptimization, using Code V for example, the lens designer can create aspecial user defined function or operand for the transverse component(e.g., ray height) of spherical aberration of the entrance pupil, whichcan then be used in a variety of ways. For example, an operand value canbe calculated as a residual sum of squares (RSS) of values across thewhole FOV or across a localized field, using either uniform ornon-uniform weightings on the field operands. In the latter case oflocalized field preferences, the values can be calculated for a locationat or near the entrance pupil, or elsewhere within a low parallax volume188, depending on the preference towards paraxial, mid, or peripheralfields. An equivalent operand can be a width of a circle of leastconfusion in a plane, such as the plane of offset NP point 192A or thatof offset NP 192B, as shown in FIG. 5C. The optimization operand canalso be calculated with a weighting to reduce or limit parallax errornon-uniformly across fields, with a disproportionate weighting favoringperipheral or edge fields over mid-fields. Alternately, the optimizationoperand can be calculated with a weighting to provide a nominally lowparallax error in a nominally uniform manner across all fields (e.g.,within or across a Core FOV 205, as in FIG. 7). That type ofoptimization may be particularly useful for mapping applications.

Whether the low-parallax lens design and optimization method usesoperands based on chief rays or spherical aberration of the entrancepupil, the resulting data can also be analyzed relative to changes inimaging perspective. In particular, parallax errors versus field andcolor can also be analyzed using calculations of the Center ofPerspective (COP), which is a parameter that is more directly relatableto visible image artifacts than is a low parallax volume, and which canbe evaluated in image pixel errors or differences for imaging objects attwo different distances from a camera system. The center of perspectiveerror is essentially the change in a chief ray trajectory given multipleobject distances—such as for an object at a close distance (3 ft),versus another at “infinity.”

In drawings and architecture, perspective, is the art of drawing solidobjects on a two-dimensional surface so as to give a correct impressionof their height, width, depth, and position in relation to each otherwhen viewed from a particular point. For example, for illustrations withlinear or point perspective, objects appear smaller as their distancefrom the observer increases. Such illustrated objects are also subjectto foreshortening, meaning that an object's dimensions along the line ofsight appear shorter than its dimensions across the line of sight.Perspective works by representing the light that passes from a scenethrough an imaginary rectangle (realized as the plane of theillustration), to a viewer's eye, as if the viewer were looking througha window and painting what is seen directly onto the windowpane.

Perspective is related to both parallax and stereo perception. In astereoscopic image capture or projection, with a pair of adjacentoptical systems, perspective is a visual cue, along with dual viewparallax, shadowing, and occlusion, that can provide a sense of depth.As noted previously, parallax is the visual perception that the positionor direction of an object appears to be different when viewed fromdifferent positions. In the case of image capture by a pair of adjacentcameras with at least partially overlapping fields of view, parallaximage differences are a cue for stereo image perception, or are an errorfor panoramic image assembly.

To capture images with an optical system, whether a camera or the humaneye, the optical system geometry and performance impacts the utility ofthe resulting images for low parallax (panoramic) or high parallax(stereo) perception. In particular, for an ideal lens, all the chiefrays from object space point exactly towards the center of the entrancepupil, and the entrance pupil is coincident with the center ofperspective (COP) or viewpoint center for the resulting images. Thereare no errors in perspective or parallax for such an ideal lens.

But for a real lens, having both physical and image quality limitations,residual parallax errors can exist. As stated previously, for a reallens, a projection of the paraxial chief rays from the first lenselement, will point towards a common point, the entrance pupil, and itslocation can be determined as an axial distance from the front surfaceof that first element. Whereas, for a real lens capturing a FOV largeenough to include non-paraxial chief rays, the chief rays in objectspace can point towards a common location or volume near, but typicallyoffset from, the center of the entrance pupil. These chief rays do notintrinsically coincide at a single point, but they can be directedthrough a small low parallax volume 188 (e.g., the LP “smudge”) byappropriate lens optimization. The longitudinal or axial variation ofrays within the LP smudge can be determined from the position a chiefray crosses the optic axis. The ray errors can also be measured as atransverse width or axial position of the chief rays within an LPsmudge.

The concept of parallax correction, with respect to centers ofperspective, is illustrated in FIG. 5D. A first camera lens 120Acollects and images light from object space 105 into at least a CoreFOV, including light from two outer ray fans 179A and 179B, whose chiefray projections converge towards a low parallax volume 188A. These rayfans can correspond to a group of near edge or edge of field rays 172,as seen in FIG. 2B or FIG. 5C. As was shown in FIG. 5C, within an LPvolume 188, the vectoral projection of such rays from object space,generally towards image space, can cross the optical axis 185 beyond theimage plane, at or near an alternate NP point 192B that can be selectedor preferred because it favors edge of field rays. However, as is alsoshown in FIG. 5C, such edge of field rays 172 need not cross the opticalaxis 185 at exactly the same point. Those differences, when translatedback to object space 105, translate into small differences in theparallax or perspective for imaged ray bundles or fans within or acrossan imaged FOV (e.g., a Core FOV 205, as in FIG. 7) of a camera lens.

A second, adjacent camera lens 120B, shown in FIG. 5D, can provide asimilar performance, and image a fan of chief rays 170, including rayfan 179C, from within a Core FOV 205 with a vectoral projection of thesechief rays converging within a corresponding low parallax volume 188B.LP volumes 188A and 188B can overlap or be coincident, or be offset,depending on factors including the camera geometries and the seamsbetween adjacent cameras, or lens system fabrication tolerances andcompensators, or on whether the device center 196 is offset from the LPvolumes 188. The more overlapped or coincident these LP volumes 188 are,the more overlapped are the centers of perspective of the two lenssystems. Ray Fan 179B of camera lens 120A and ray fan 179C of cameralens 120B are also nominally parallel to each other; e.g., there is noparallax error between them. However, even if the lens designs allowvery little residual parallax errors at the FOV edges, fabricationvariations between lens systems can increase the differences.

Analytically, the chief ray data from a real lens can also be expressedin terms of perspective error, including chromatic errors, as a functionof field angle. Perspective error can then be analyzed as a positionerror at the image between two objects located at different distances ordirections. Perspective errors can depend on the choice of COP location,the angle within the imaged FOV, and chromatic errors. For example, itcan be useful to prioritize a COP so as to minimize green perspectiveerrors. Perspective differences or parallax errors can be reduced byoptimizing a chromatic axial position (Dz) or width within an LP volume188 related to a center of perspective for one or more field angleswithin an imaged FOV. The center of perspective can also be graphed andanalyzed as a family of curves, per color, of the Z (axial) interceptposition (distance in mm) versus field angle. Alternately, to get abetter idea of what a captured image will look like, the COP can begraphed and analyzed as a family of curves for a camera system, as aparallax error in image pixels, per color, versus field.

During the design or a camera lens systems, a goal can be to limit theparallax error to a few pixels or less for imaging within a Core FOV 205(FIG. 7). Alternately, it can be preferable to particularly limitparallax errors in the peripheral fields, e.g., for the outer edges of aCore FOV and for an Extended FOV region (if provided). If the residualparallax errors for a camera are thus sufficiently small, then theparallax differences seen as a perspective error between two adjacentcameras near their shared seam 160, or within a seam related region ofextended FOV overlap imaging, can likewise be limited to several pixelsor less (e.g., ≤3-4 pixels). Depending on the lens design, devicedesign, and application, it can be possible and preferable to reduceparallax errors for a lens system further, as measured by perspectiveerror, to ≤0.5 pixel for an entire Core FOV, the peripheral fields, orboth. If these residual parallax errors for each of two adjacent camerasare small enough, images can be acquired, cropped, and readily tiled,while compensating for or hiding image artifacts from any residual seams160 or blind regions 165.

In pursuing the design of a panoramic camera of the type of that of FIG.1, but to enable an improved low-parallax multi-camera panoramic capturedevice (300), having multiple adjacent cameras, the choices of lensoptimization methods and parameters can be important. A camera lens 120,or system of lens elements 135, like that of FIG. 2A, can be used as astarting point. The camera lens has compressor lens element(s), andinner lens elements 140, the latter of which can also be defined asconsisting of a pre-stop wide angle lens group, and a post-stopeyepiece-like lens group. In designing such lenses to reduce parallaxerrors, it can be valuable to consider how a fan of paraxial tonon-paraxial chief rays 125 (see FIG. 2A), or a fan of edge chief rays170 (see FIG. 2B), or localized collections of edge of field rays 172(see FIG. 5C) or 179 A,B (see FIG. 5D) are imaged by a camera lensassembly. It is possible to optimize the lens design by using a set ofmerit function operands for a collection or set (e.g., 31 defined rays)of chief rays, but the optimization process can then become cumbersome.As an alternative, in pursuing the design of an improved low-parallaxmulti-camera panoramic capture device (300), it was determined thatimproved performance can also be obtained by using a reduced set of rayparameters or operands that emphasizes the transverse component ofspherical aberration at the entrance pupil, or at a similar selectedsurface or location (e.g., at an offset NP point 192A or 192B) within anLP smudge volume 188 behind the lens system. Optimization for atransverse component of spherical aberration at an alternatenon-paraxial entrance pupil can be accomplished by using merit functionweightings that emphasize the non-paraxial chief rays.

As another aspect, in a low-parallax multi-camera panoramic capturedevice, the fans of chief rays 170 that are incident at or near abeveled edge of an outer lens element of a camera 120 (see FIG. 2B)should be parallel to a fan of chief rays 170 that are incident at ornear an edge 132 of a beveled surface of the outer lens element of anadjacent camera (see FIG. 1). It is noted that an “edge” of an outerlens element 137 or compressor lens is a 3-dimensional structure (seeFIG. 2B), that can have a flat edge cut through a glass thickness, andwhich is subject to fabrication tolerances of that lens element, theentire lens assembly, and housing 130, and the adjacent seam 160 and itsstructures. The positional definition of where the beveled edges are cutinto the outer lens element depends on factors including the materialproperties, front color, distortion, parallax correction, tolerances,and an extent of any extra extended FOV 215. An outer lens element 137becomes a faceted outer lens element when beveled edges 132 are cut intothe lens, creating a set of polygonal shaped edges that nominally followa polygonal pattern (e.g., pentagonal or hexagonal).

A camera system 120 having an outer lens element with a polygonal shapethat captures incident light from a polygonal shaped field of view canthen form a polygonal shaped image at the image plane 150, wherein theshape of the captured polygonal field of view nominally matches theshape of the polygonal outer lens element. The cut of these bevelededges for a given pair of adjacent cameras can affect both imaging andthe optomechanical construction at or near the intervening seam 160.

As another aspect, FIG. 5E depicts “front color”, which is a differencein the nominal ray paths by color versus field, as directed to an offaxis or edge field point. Typically, for a given field point, the bluelight rays are the furthest offset. As shown in FIG. 5E, the acceptedblue ray 157 on a first lens element 137 is DX≈1 mm further out than theaccepted red ray 158 directed to the same image field point. If the lenselement 137 is not large enough, then this blue light can be clipped orvignetted and a color shading artifact can occur at or near the edges ofthe imaged field. Front color can appear in captured image content as anarrow rainbow-like outline of the polygonal FOV or the polygonal edgeof an outer compressor lens element 437 which acts as a field stop forthe optical system. Localized color transmission differences that cancause front color related color shading artifacts near the image edgescan be caused by differential vignetting at the beveled edges of theouter compressor lens element 137, or from edge truncation at compressorlens elements 438 (FIG. 13A), or through the aperture stop 145. Duringlens design optimization to provide an improved camera lens (320), frontcolor can be reduced (e.g., to DX≤0.5 mm width) as part of the chromaticcorrection of the lens design, including by glass selection within thecompressor lens group or the entire lens design, or as a trade-off inthe correction of lateral color. The effect of front color on capturedimages can also be reduced optomechanically, by designing an improvedcamera lens (320) to have an extended FOV 215 (FIG. 7), and also theopto-mechanics to push straight cut or beveled lens edges 132 at orbeyond the edge of the extended FOV 215, so that any residual frontcolor occurs outside the core FOV 220. The front color artifact can thenbe eliminated during an image cropping step during image processing. Theimpact of front color or lateral color can also be reduced by aspatially variant color correction during image processing. As anotheroption, an improved camera lens (320) can have a color dependentaperture at or near the aperture stop, that can, for example, provide alarger transmission aperture (diameter) for blue light than for red orgreen light.

Optical performance at or near the seams can be understood, in part,relative to distortion (FIG. 6) and a set of defined fields of view(FIG. 7). In particular, FIG. 7 depicts potential sets of fields of viewfor which potential image light can be collected by two adjacentcameras. As an example, a camera with a pentagonally shaped outer lenselement, whether associated with a dodecahedron or truncated icosahedronor other polygonal lens camera assembly, with a seam 160 separating itfrom an adjacent lens or camera channel, can image an ideal FOV 200 thatextends out to the vertices (60) or to the polygonal edges of thefrustum or conical volume that the lens resides in. However, because ofthe various physical limitations that can occur at the seams, includingthe finite thicknesses of the lens housings, the physical aspects of thebeveled lens element edges, mechanical wedge, and tolerances, a smallercore FOV 205 of transiting image light can actually be imaged. Thecoated clear aperture for the outer lens elements 137 should encompassat least the core FOV 205 with some margin (e.g., 0.5-1.0 mm). As thelens can be fabricated with AR coatings before beveling, the coatingscan extend out to the seams. The core FOV 205 can be defined as thelargest low parallax field of view that a given real camera 120 canimage. Equivalently, the core FOV 205 can be defined as the sub-FOV of acamera channel whose boundaries are nominally parallel to the boundariesof its polygonal cone (see FIGS. 5A and 5B). Ideally, with small seams160, and proper control and calibration of FOV pointing, the nominalCore FOV 205 approaches or matches the ideal FOV 200 in size.

During a camera alignment and calibration process, a series of imagefiducials 210 can be established along one or more of the edges of acore FOV 205 to aid with image processing and image tiling or mosaicing.The resulting gap between a core FOV 205 supported by a first camera andthat supported by an adjacent camera can result in blind regions 165(FIG. 5A, B). To compensate for the blind regions 165, and theassociated loss of image content from a scene, the cameras can bedesigned to support an extended FOV 215, which can provide enough extraFOV to account for the seam width and tolerances, or an offset devicecenter 196. As shown in FIG. 7, the extended FOV 215 can extend farenough to provide overlap 127 with an edge of the core FOV 205 of anadjacent camera, although the extended FOVs 215 can be larger yet. Thislimited image overlap can result in a modest amount of image resolutionloss, parallax errors, and some complications in image processing aswere previously discussed with respect to FIG. 3, but it can also helpreduce the apparent width of seams and blind regions. However, if theextra overlap FOV is modest (e.g., ≤5%) and the residual parallax errorstherein are small enough (e.g. ≤0.75 pixel perspective error), asprovided by the present approach, then the image processing burden canbe very modest. Image capture out to an extended FOV 215 can also beused to enable an interim capture step that supports camera calibrationand image corrections during the operation of an improved panoramicmulti-camera capture device 300. FIG. 7 also shows an inscribed circlewithin one of the FOV sets, corresponding to a subset of the core FOV205, that is the common core FOV 220 that can be captured in alldirections from that camera. The angular width of the common core FOV220 can be useful as a quick reference for the image capacity of acamera. An alternate definition of the common core FOV 220 that islarger, to include the entire core FOV 205, can also be useful. Thedashed line (225) extending from the common core FOV 220 or core FOV205, to beyond the ideal FOV 200, to nominally include the extended FOV215, represents a region in which the lens design can support carefulmapping of the chief or principal rays or control of sphericalaberration of the entrance pupil, so as to enable low-parallax errorimaging and easy tiling of images captured by adjacent cameras.

Across a seam 160 spanning the distance between two adjacent usableclear apertures between two adjacent cameras, to reduce parallax andimprove image tiling, it can be advantageous if the image light iscaptured with substantial straightness, parallelism, and common spacingover a finite distance. The amount of FOV overlap needed to provide anextended FOV and limit blind regions can be determined by controllingthe relative proximity of the entrance pupil (paraxial NP point) or analternate preferred plane within a low parallax volume 188 (e.g., toemphasize peripheral rays) to the device center 196 (e.g., to the centerof a dodecahedral shape). The amount of Extended FOV 215 is preferably5% or less (e.g., ≤1.8° additional field for a nominal Core FOV of37.5°), such that a camera's peripheral fields are then, for example,˜0.85-1.05). If spacing constraints at the device center, andfabrication tolerances, are well managed, the extended FOV 215 can bereduced to ≤1% additional field. Within an extended FOV 215, parallaxshould be limited to the nominal system levels, while both imageresolution and relative illumination remain satisfactory. The parallaxoptimization to reduce parallax errors can use either chief ray or pupilaberration constraints, and targeting optimization for a high FOV region(e.g., 0.85-1.0 field), or beyond that to include the extra cameraoverlap regions provided by an extended FOV 215 (e.g., FIG. 7, afractional field range of ˜0.85-1.05).

In addition, in enabling an improved low-parallax multi-camera panoramiccapture device (300), with limited parallax error and improved imagetiling, it can be valuable to control image distortion for image lighttransiting at or near the edges of the FOV, e.g., the peripheral fields,of the outer lens element. In geometrical optics, distortion is adeviation from a preferred condition (e.g., rectilinear projection) thatstraight lines in a scene remain straight in an image. It is a form ofoptical aberration, which describes how the light rays from a scene aremapped to the image plane. In general, in lens assemblies used for imagecapture, for human viewing it is advantageous to limit image distortionto a maximum of +/−2%. In the current application, for tiling orcombining panoramic images from images captured by adjacent cameras,having a modest distortion of ≤2% can also be useful. As a reference, inbarrel distortion, the image magnification decreases with distance fromthe optical axis, and the apparent effect is that of an image which hasbeen mapped around a sphere (or barrel). Fisheye lenses, which are oftenused to take hemispherical or panoramic views, typically have this typeof distortion, as a way to map an infinitely wide object plane into afinite image area. Fisheye lens distortion (251) can be large (e.g., 15%at full field or 90° half width (HW)), as a deviation from f-thetadistortion, although it is only a few percent for small fields (e.g.,≤30° HW). As another example, in laser printing or scanning systems,f-theta imaging lenses are often used to print images with minimalbanding artifacts and image processing corrections for pixel placement.In particular, F-theta lenses are designed with a barrel distortion thatyields a nearly constant spot or pixel size, and a pixel positioningthat is linear with field angle θ, (h=f*θ).

Thus, improved low-parallax cameras 320 that capture half FOVs of≤35-40° might have fisheye distortion 251, as the distortion may be lowenough. However, distortion can be optimized more advantageously for thedesign of improved camera lens assemblies for use in improvedlow-parallax multi-camera panoramic capture devices (300). As a firstexample, as shown in FIG. 6, it can be advantageous to provide cameralens assemblies with a localized nominal f-theta distortion 250A at ornear the edge of the imaged field. In an example, the image distortion250 peaks at ˜0.75 field at about 1%, and the lens design is notoptimized to provide f-theta distortion 250 below ˜0.85 field. However,during the lens design process, a merit function can be constrained toprovide a nominally f-theta like distortion 250A or an approximatelyflat distortion 250B, for the imaged rays at or near the edge of thefield, such as for peripheral fields spanning a fractional field rangeof ˜0.9-1.0. This range of high fields with f-theta type or flatteneddistortion correction includes the fans of chief rays 170 or perimeterrays of FIG. 2B, including rays imaged through the corners or vertices60, such as those of a lens assembly with a hexagonal or pentagonalouter lens element 137. Additionally, because of manufacturingtolerances and dynamic influences (e.g., temperature changes) that canapply to a camera 120, including both lens elements 135 and a housing130, and to a collection of cameras 120 in a panoramic multi-cameracapture device, it can be advantageous to extend the region of nominalf-theta or flattened distortion correction in peripheral fields tobeyond the nominal full field (e.g., 0.85-1.05). This is shown in FIG.6, where a region of reduced or flattened distortion extends beyond fullfield to ˜1.05 field. In such a peripheral field range, it can beadvantageous to limit the total distortion variation to ≤0.5% or less.Controlling peripheral field distortion keeps the image “edges” straightin the adjacent pentagonal shaped regions. This can allow more efficientuse of pixels when tiling images, and thus faster image processing.

The prior discussion treats distortion in a classical sense, as an imageaberration at an image plane. However, in low-parallax cameras, thisresidual distortion is typically a tradeoff or nominal cancelation ofcontributions from the compressor lens elements (137, or 437 and 438 inFIG. 13A) versus those of the aggregate inner lens elements (140, or 440in FIG. 13A). Importantly, the ray re-direction caused by the distortioncontribution of the outer compressor lens element also affects both theimaged ray paths and the projected chief ray paths towards the lowparallax volume. This in turn means that for the design of at least somelow-parallax lenses, distortion optimization can affect parallax or edgeof field NP point or center of perspective optimization.

The definitions of the peripheral fields or a fractional field range 225of (e.g., ˜0.85-1.05, or including ≤5% extra field), in which parallax,distortion, relative illumination, resolution, and other performancefactors can be carefully optimized to aid image tiling, can depend onthe device and camera geometries. As an example, for hexagonal shapedlenses and fields, the lower end of the peripheral fields can be definedas ˜0.83, and for pentagonal lenses, ˜0.8. Although FIG. 7 wasillustrated for a case with two adjacent pentagon-shaped outer lenselements and FOV sets, the approach of defining peripheral fields andExtended FOVs to support a small region of overlapped image capture, canbe applied to multi-camera capture device designs with adjacentpentagonal and hexagonal cameras, or to adjacent hexagonal cameras, orto cameras with other polygonal shapes or with adjacent edges of anyshape or contour generally.

For an Extended FOV 215 to be functionally useful, the nominal imageformed onto an image sensor that corresponds to a core FOV 205 needs tounderfill the used image area of the image sensor, by at least enough toallow an extended FOV 215 to also be imaged. This can be done to helpaccount for real variations in fabricated lens assemblies from theideal, or for the design having an offset device center 196, as well asfabrication variations in assembling an improved low-parallaxmulti-camera panoramic capture device (300). But as is subsequentlydiscussed, prudent mechanical design of the lens assemblies can impactboth the imaged field of view of a given camera and the seams betweenthe cameras, to limit mechanical displacements or wedge and help reduceparallax errors and FOV overlap or underlap. Likewise, tuning the imageFOV (core FOV 205) size and position with compensators or with fiducialsand image centroid tracking and shape tracking can help. Taken togetherin some combination, optimization of distortion and low or zero parallaximaging over extended peripheral fields, careful mechanical design tolimit and compensate for component and assembly variations, and the useof corrective fiducials or compensators, can provide a superior overallsystems solution. As a result, a captured image from a camera canreadily be cropped down to the nominal size and shape expected for thenominal core FOV 205, and images from multiple cameras can then bemosaiced or tiled together to form a panoramic image, with reducedburdens on image post-processing. However, an extended FOV 215, ifneeded, should provide enough extra angular width (e.g., q₁≤5% of theFOV) to match or exceed the expected wedge or tilt angle q₂, that canoccur in the seams, q₁≥q₂.

In designing an improved imaging lens of the type that can be used in alow-parallax panoramic multi-camera capture device (100 or 300), severalfirst order parameters can be calculated so as to inform the designeffort. A key parameter is the target size of the frustum or conicalvolume, based on the chosen polygonal configuration (lens size (FOV) andlens shape (e.g., pentagonal)) and the sensor package size. Other keyparameters that can be estimated include the nominal location of theparaxial entrance pupil, the focal lengths of the compressor lens groupand the wide-angle lens group, and the FOV seen by the wide-angle group.

But the design optimization for an improved camera lens (320) for use inan improved low-parallax panoramic multi-camera capture devices (300)also depends on how the numerous other lens attributes and performancemetrics are prioritized. In particular, the relevant system parameterscan include the control of parallax or the center of perspective (COP)error at the edges of an imaged field or for inner field locations orboth, as optimized using fans of chief rays or spherical aberration ofthe entrance pupil). These parameters are closely linked with other keyparameters including the width and positions of the “LP smudge” orvolume 188, the size of any center offset distance between the entrancepupil or LP smudge and the device center 196, the target width of thegaps or seams, the extent of blind regions 165, and the size of anymarginal or extended FOV to provide overlap. The relevant performancemetrics can include image resolution or MTF, distortion (particularly inthe peripheral fields, and distortion of the first compressor lenselement and of the compressor lens group), lateral color, relativeillumination, front color, and color vignetting, telecentricity, andghosting. Other relevant design variables can include mechanical andmaterials parameters such as the number of compressor lens elements, theconfiguration of the compressor lens group, the wide-angle lens groupand eyepiece lens group, glass choices, the allowed maximum size of thefirst compressor or outer lens element, the sensor package size, thetrack length, the nominal distance from the image plane to the nearestprior lens element (e.g., working distance), the nominal distance fromthe image plane to the entrance pupil, the nominal distance from theimage plane or the entrance pupil to the polygonal center or devicecenter, manufacturing tolerances and limits, and the use ofcompensators.

FIG. 13A provides a cross-sectional view of an alternate and improvedopto-mechanical design for an improved camera 320 that can be used in animproved panoramic multi-camera capture device 300. In this exemplarydesign, the camera lens 320 has a lens form in which the compressor lenselement has been split into a compressor lens group, including first andsecond compressor lens elements (437 and 438). The inner lens elements400 include a wide-angle lens group located prior to the aperture stop445, that includes a fourth lens element 442, and a post-stop eyepiecelens group. As previously, the lens system provides a paraxial NP point490 that is offset from a non-paraxial chief ray NP point 492 that liewithin a low parallax volume 488. The size or width of this volume, andthe location of NP Points of potential interest (e.g., paraxial, midfield, peripheral field, circle of least confusion based) within it,depends on design priorities and parallax optimization (e.g., sphericalaberration of the entrance pupil, chief ray fans, distortion). While thelens form of the example camera lens systems designs of FIG. 2A and FIG.13A are similar, the lenses vary in details and performance, includingwith their different compressor lens configurations. Because ofdifferences in specifications and optimization methods and priorities,these lenses are also different in cost, performance, andmanufacturability.

Depending on priorities, these lens systems can be optimized further,and the different variations of the lens form may be individually bettersuited for different markets or applications. In general, the outermostlens element, or first compressor lens element, has used Ohara SLAH52,SLAH53, and SLAH63 glasses (or equivalent glasses from Schott Glass(e.g., N-LAF36 and N-LASF43)), which are high index, low dispersionflint glasses with visible spectra refractive indices n˜1.80 and an Abbenumber Vd ˜41.5. It should be understood that other optical materialscan be used for the lens elements in the camera lenses 520 generally,including for the compressor lens elements. For example, use of a highindex, lower dispersion, mid-crown glass like Ohara SLAL-18 can behelpful for color correction. As another example, lens elements can alsobe made from optical ceramics such as Alon (n˜1.79, Vd˜57-72) or Spinel,which are extremely durable materials, similar to sapphire, but withexcellent optical transparency, low dispersion, and a controllablymodifiable isotropic crystalline structure. It should also be understoodthat the camera lenses of the present approach can also be designed withoptical elements that consist of, or include, refractive, gradientindex, glass or optical polymer, reflective, aspheric or free-form,Kinoform, fresnel, diffractive or holographic, sub-wavelength ormetasurface, optical properties. These lens systems can also be designedwith achromatic or apochromatic color correction, or with thermaldefocus desensitization.

Although FIG. 13A does not depict how the improved camera lens can bemounted within a lens housing, the close proximity of the large outercompressor lens element 437 to the large second compressor lens elements438 that form a doublet, can require a careful to robustly support theseelements in close proximity. As one approach, the first compressor lenselement 437 can be positionally centered to the compressor doublet(438), and the doublet can be centered to a primary circular datum onthe inside surface of the lens housing. This datum can be tightlytolerance to a channel centering hub 330, so as to reduce tolerancebuildup between adjacent camera channels.

While improving the optical design of the camera lens systems isimportant for enabling improved low-parallax panoramic multi-cameracapture devices (300), improving the opto-mechanical design can beequivalently important. As suggested previously, the actual performanceof a camera 120 can vary from the designed performance due to materialsand fabrication variations amongst the individual lens elements 135 andthe housing 130 and its constituent components, and the interactionsthereof. As a result of such variations, the image quality (e.g.,aberrations, including distortion), focal length (EFL) andmagnification, working distance or track length, beam pointing or imagelocation, and other attributes of a camera 120 can vary. Thesevariations also mean that the assembly and performance of a given cameravaries from that of another camera with nominally the identicalopto-mechanical design. For example, the focal length of a set ofnominally identical cameras can vary by ±2%, which in turn will cause asimilar variation in the lens magnifications and FOVs. This variationcan be reduced or eliminated by designing improved camera lensesvarifocally to include a focal length compensator, such as with a lenselement whose axial position can be adjusted. Alternately, the cameras120 can be designed such that nominal image from a nominal cameraunderfills the image sensor, enough so that the image from a camera witha large (e.g., +2%) focal length lens also underfills a sensor, albeitwith less margin. During calibration to determine a FOV, the EFL ormagnification of a lens can be measured, and the sensor can also bealigned to be in focus for that lens. Image processing software can thenbe used to compensate the image for the lens variations, includingcompensating image size for magnification and distortion variationsbetween lenses.

Considering the opto-mechanics in greater detail, the axial alignment orfocal position of an image sensor, relative to an image plane 150provided by a camera lens assembly (320) can be improved by means of anappropriate mechanism. For example, FIG. 8 and FIG. 9 show a portion ofan improved panoramic multi-camera capture device 300, in which an imagesensor 270 can be assembled into a sensor package 265 that includes amount 275 which includes a plate 290, a circular flange, severaladjustment screws 280, flexures, or springs 285. For example, threeadjustment screws can be used to control X translation along with Z-axisrotation, and another set of three screws can be used to control Z-axistranslation along with X-Y axes rotations, and an additional screw isused to control Y axis translation. A pair of springs are used to retaina gimbal plate 290, while also allowing X and Z-axis translationsrespectively. Other adjustment designs or devices can be used, such asusing pins and micrometers or pins and shims, within the tight spaceconstraints that the camera 320 and the overall panoramic multi-cameracapture device 300 allows.

Fabrication variations for individual cameras, and the opto-mechanicalinteractions between them, can have significant impact on the design andperformance of a multi-camera capture device 300. During the initialassembly of a multi-camera capture device 300, the interaction oftolerances and mechanical wedge between a housing 430 of a first camera320 and a housing 430 of a second adjacent and abutting camera 320 canaffect the seams 400, the pointing of the Core FOV 205 or Extended FOV215 of individual cameras (causing FOV overlap or underlap), and thusaffect a FOV captured by each of the cameras. Furthermore, co-alignmentmounting stresses imparted to adjacent cameras by the close proximitymounting can physically distort one or more of the camera housings, andthen potentially also distort or skew the optical imaging function ofthe camera lenses. As a result, a camera lens system can provide animage to the sensor that is shifted or rotated or tilted out of plane.These potential problems, or the risk of their occurrence, can beexacerbated by environmental influences, such as asymmetrical thermalloading or substantial asymmetrical optical loading.

To counter such issues, an improved multi-camera capture device 300, asshown in FIG. 9, can include features to provide kinematic type mountingof individual cameras 320 or objective lenses. In particular, FIG. 9depicts two views of a dodecahedron multi-camera capture device 300,including a partial cross-section in which 11 pentagonal cameras 320 aremounted to a central support 325 that occupies the nominal position of atwelfth potential camera channel. Each camera 320 has a separate baselens assembly or housing 430 that consists of a lens mount which mountsthe compressor lens (437) while also mounting the inner lens elements440 that together comprise a base lens assembly. Although for eachcamera 320, the lens elements and housings 430 fit within the nominalconical space or volume, they need not nominally fill that space.Indeed, the abrupt ray bending provided by the compressor lens elementscan mean that the inner lens elements 440 and their housings or barrelsunderfill the available space, and the overall lens housings 430 cantaper further inwards, potentially leaving an open inner volume 390between adjacent lens assemblies.

The housings 430 or base lens assemblies of FIG. 9 also include a turnedsection, that can be machined on a CNC multi axis (5-axis) machine, andthat mates with a tripod-like channel centering hub 330. The channelcentering hubs 330 can be entirely turned on a lathe except for thepentagonal flange, which is completed in a finish operation after thelathe. Being turned on a lathe means that exceptional concentricity andrunout can be achieved, helping with the ultimate alignment of thechannel. The housing 430 mates with the inside diameter of the channelcentering hub 330 which is a key part of a central mount mechanicalassembly that is designed to have a fit with it that ranges from a slipfit to a light interference fit, so as to ensure axial alignment withoutsignificant variations due to gap tolerances. This same fit reducesperpendicularity errors with respect to the channel axis.

The tripods or channel centering hubs 330 also include a turned sectionor ball pivot 340 that mates with a socket 345 of a spherical socketarray 346 provided on the central support 325. In this system, thecamera 320 located in the polar position, opposite the central support325, is a rigidly placed reference channel. The center support 325consists of a cylindrically shaped post with a ball on the top. Thegeometries for the center support 325 and tripods or centering hubs 330can be designed to provide more space for power and communicationscables, cooling lines, and mechanisms to secure the lens housings 430 orcables. In this example, the ball contains sockets 345, each of whichcan receive a ball pivot 340. The ball pivots 340 are at the ends ofextended pins or ball pivot arms 342. Although this ball and socketportion of the center support 325 mount can be expensive to machine,given the precision expected with respect to the position and depth ofthe sockets, the advantages are that centerline pointing is controlled,while there is only a single part per device 300 that demandsexceptional precision. Whereas, each of the camera channels 320 may bemachined with less precision, which eases both the fabrication andreplacement costs.

The individual camera lens housings 430 of FIG. 9 can also be providedwith external or outside channel to channel datums 335, located midwayalong the pentagonal sides. Each of these channel-to-channel datums 335can comprise two parallel convexly curved slightly protruding bars thatare separated by an intervening groove. These datums are designed toprovide both single point or localized kinematic contacts orinteractions between lens housings, such that the datum featuresinterweave in such a way that only one part or housing will dominate interms of tolerance. Since they are interwoven, only the variation of onepart will influence the distance between each camera channel, and thusinfluence the angle between the channels. In particular, if one datum335 is larger it will dominate because the other will not make contact.Thus, only one tolerance contributes for two parts. That the channel tochannel datums 335 are interwoven from one camera 320 to another, alsolimits lateral movement between mating (pentagonal) faces or sides,while allowing limited angular movement of the lens housings 430.

Individually, and in aggregate, the interactions between camera lenshousings 430 or base lens assembly's limits mechanical displacements andwedge or channel pointing errors (roll, pitch and yaw) between camerasdue to both the ball and socket arrangement and the datum features(335). Each camera channel assembly works together with its neighbors tolimit channel pointing error. The portion of the base lens assembly(430) that holds the outer lens element 437 or compressor lens also hasinternal functional datums that can locate the compressor lensperpendicular relative, to the optical or mechanical channel axis, andit has additional internal datum features that limit axial misalignment.As will be discussed in subsequent detail with respect to FIG. 16, anadditional set of alignment features provided on the ground edges of thecompressor lens can function as a datum and interact with these internaldatum features to limit pentagonal rotation about the channel axis aswell.

The use of the alignment features depicted in FIG. 9, and particularlythe ball pivot and socket datums (350 and 356) and the channel tochannel datum features 335 reduces the risks of rotation, pivoting, orsplay from one camera channel (320) to another. Thus, these featuresalso help enable the seams 400 between cameras 320 to have moreconsistent thicknesses, with respect to the design values, than may havehappened otherwise. The use of the internal features within the lenshousing (e.g., compensators, adjustments screws, and shims) and externalfeatures between lens housings (e.g., channel to channel datums, balland socket datums, and a channel loading support) help control Core FOVor Extended FOV pointing, so that one camera channel can be aligned toanother adjacent channel. The combined use of channel to channel datums,ball and socket datums (FIG. 10), and a channel loading support (FIG.17) also can help desensitize the device to mechanical or thermal loads.

FIG. 10 depicts a further design option for improving theopto-mechanical design of the outer lens elements 437, or compressorlens elements, in proximity to the seams 400 in an improved multi-camerapanoramic image capture device 300. In particular, the edges of an outerlens element 437 can protrude beyond the upper or outer edge of a lenshousing 430 such that two adjacent outer lens elements 437 of adjacentcameras 320 can be in near contact at the seams 400. If these outer lenselements 437 are fabricated at least in part with a somewhat compliantmaterial, then some amount of actual physical contact can be allowed.If, however, these outer lens elements 437 are fabricated with a brittlematerial, such as glass, then greater care is required.

In the configuration of FIG. 10, two adjacent outer lens elements 437that have protruding edges are in near contact at the seams 400. In apreferred design approach, the opposing edges are provided with astepped edge angle 365 or structure. In the outermost portion, the twolenses and housing can provide a parallel seam 400 with a width of 1.0mm or smaller.

In the innermost portion, where the inner beveled edges 370 of theadjacent outer lens elements 437 approach each other, the lens housing430, channel to channel datums 335, and flat surface datums 670 can beprovided. The edge of each outer lens element 437 then has a steppededge groove 380 that can be filled with a compliant adhesive. Abovethat, along the edges 432 or seams 400, spanning the outermost edgeportion of the adjacent outer lens elements 437, these lens elements canbe nearly abutting, and separated by a gap that can be only 0.5 mm wide,or smaller. In practice, the optimization of the seam width can dependon how brittle or compliant the lens material is (glass or polymer), theflexibility of a seam filling adhesive, the use of other protectivemeasures, and the application.

FIG. 17 provides greater detail on how these and other features can beused during alignment and assembly. In particular, FIG. 17 depicts across-sectional view of a portion of an improved multi-camera panoramicimage capture device 300 of the present approach, in which 11 camerachannels 320 are attached to a post or vertical central support 325through which both wiring and cooling can be provided (see FIG. 15 formore details). FIG. 17 in particular depicts a design for theopto-mechanical hardware, albeit assembled without the lens elements(see FIGS. 9 and 15 for illustrations of opto-mechanical hardwareincluding lens elements). FIG. 18A depicts key elements or componentsused in this example alignment and assembly approach in yet greaterdetail.

As shown in FIG. 17, a top camera 320 can be identified as a primaryalignment channel 610, while all other camera channels 320 shown areidentified as secondary channels 615. In particular, for the primarychannel, the ball pivot arm 342 with ball pivot 340 of the channelcentering hub 330, mates into a ball socket datum 356 of the ball socketarray 346, where it can interact with a locking axial retention pin 348that slips into an indent between the ball pivot arm 342 and the ballpivot 340, to stop Z-axis (vertical) translation. An anti-rotationkeying pin 349 is press fit into a localized hole provided in the ballpivot arm, so as to stop rotation of the primary camera channel 610about the z-axis. The ball pivot arm 342 of the primary camera channel610 has a light press fit into its ball socket datum 356, so as toprevent rotations about the X and Y axes.

Alternately, the ball sockets 345 can have a latch mechanism (not shown)to load a ball pivot 340 against a ball socket datum 356, as a means toprovide both a loading force and a mechanism that has reducedsensitivity to a mechanical loading force being applied to the camera orthe device. For example, a latching mechanism can have latches actuatedwith linkage assemblies. As compared to using a retention pin, alatching mechanism can be more compliant, robust, and reliable, if acamera channel 320 or the device 300 are impacted by a loading force. Itis also noted that the lens housings 430 can also have identifying marksto ease alignment of a housing with adjacent lens housings.

As previously shown in FIG. 9, and shown in yet greater detail in FIG.18A, each camera channel 320 has a pair of peripheral channel to channeldatums 335 on each face, that are small (localized) and nominallycentered along the faces or sides of the lens housing. For a pentagonallens, each of the five sides would have a pair of datums (335). Thesedatums are provided with curved surfaces 373 (shown in bold, andextended, for emphasis) so as to enable point to point type contact(375). In particular, each pair of channel to channel datums 335 hasside faces which act as lateral datums between camera channels. They arecurved to accommodate relative angular movement between channels. Eachside datum only has an effective single point of contact 375 where eachof their mating datum radial surfaces meet. Since the radii are verylarge, the datum surfaces approach a straight line. Thus, any shiftingof the camera channels relative to each other that could result in anangular or centerline offset can then only have a minute effect on therelative lateral offset of a given camera channel.

As further shown in FIG. 17, an improved multi-camera panoramic imagecapture device 300 can also have a channel loading support 630 thatbiases all secondary channels 615 against the primary channel 610. Thechannel loading support 630 employs peripheral datum pairs that arenominally identical to the channel to channel datums 335 used on eachchannel. The channel loading support 630 can also have a spring element635 to facilitate loading of the secondary channels 615 against theprimary channel 610. The channel loading support 630 can also utilize akey feature for anti-rotation, although using a keyed support may impartsuperfluous constraints.

The primary alignment channel 610 (see FIG. 17) is aligned and lockedinto place when its ball pivot arm 342 and ball pivot 340 are engagedwith the pins (348, 349) of the channel centering hub 330. The secondarycamera channels 615 are then added, and are loosely aligned with theirball pivots 340 fit against datums within the sockets of the channelcentering hub 330. When the channel loading support 330 is added, at theseams 400, it nudges the secondary channels 615 both against the primarychannel 610 and each other. The relative size of the ball socket 345 tothe ball pivots 340 of the secondary channels are provided so that thesecondary channels 615 are constrained by the ball socket 345 in onlythe Z-direction.

Then at the seams 400, for any two faces of adjacent camera channels(610, 615) to be parallel to each other, the configuration requires thatat least three camera channels have their peripheral channel to channeldatums 335, with points of contact 375, in contact with opposing channelto channel datums 335 with their points of contact 375. This effectivelyconstrains the secondary channels 615 for three degrees of freedom(DOF). It is recognized that applying traditional kinematic mechanicaldesign principles, in which motion and constraints in 6 DOF can beprecisely limited with little crosstalk, can be tricky in a device thatcan intrinsically locate a multitude of complex polygonal faces (e.g.,hexagonal or pentagonal) in close proximity. In a case where more than 3camera channels are in close proximity, if not in contact, with eachother, the camera channels could become over constrained. Because thesystem can then be only quasi-kinematic, mechanical stresses and strainscould then cause component mis-alignment or damage. These potentialproblems can be overcome by a variety of methods. As one example, therelatively small size and centered locations of the channel to channeldatums 335 can limit any angular or spatial misalignment to relativelysmall amounts. As another example, a compliant material such as RTV canbe applied in the seams 400 to absorb some stresses or strains, whilethe lens housings 430 can be designed and fabricated to be sufficientlyrigid to resist deformations, mis-alignments, or damage. Seam widthvariations, including dynamic ones caused by thermal or mechanicalloads, can also be compensated by providing improved cameras 320 thatcapture image light 415 with an extended FOV 215.

As shown in FIG. 18A, a point of contact can occur along a line acrossthe radial surface of a datum. Thus, each channel to channel datum 335makes “line” contact with an adjacent set of channel to channel datums335 if the mating surfaces are perfectly parallel. If not, only pointcontact is made. Typically, the peripheral channel to channel datums 335only contact an opposing face at one point. For each channel to channeldatum pair, only a single datum is likely to contact due to dimensionalvariations. This is an asset of this design approach, since only asingle datum can influence the gap or seam 400 between adjacent camerachannels. This approach can be used in various designs for an improvedmulti-camera panoramic image capture device 300, including a device witha “soccer ball” or truncated icosahedron (FIG. 4) geometry, where theprimary alignment (camera) channel can be an external hexagonally shapedlens. This approach can also be used in the construction of“hemispherical” devices (see FIG. 21).

FIG. 18B-1 depicts an across seam cross-sectional view of an alternateor refined version of the construction of lens housings 430, and theirinterface near a seam 400, to that shown in FIG. 10 and FIG. 18A. Inparticular, FIG. 18B-1 depicts portions of two adjacent lens housings430, located around a seam 400, where each housing support an outercompressor lens element 437 and at least a second compressor lenselement 438. The lens housings 430 include internal light traps 457 andsides that extend up into grooves 433 that have been cut into the edges432 of the outer lens elements 437. The outer walls of the lens housings430 are tapered, expanding towards the device center, so as to providergreater mechanical rigidity and robustness. The interaction of two pairsof adjacent channel to channel datums 335 can be seen in cross-section.

FIG. 18B-2 depicts an alternate cross-sectional view of these samecomponents, but cut along or within a seam 400, to show the interactionof channel to channel datums 335. As in FIG. 18A, the adjacent pairchannel to channel datums 335 a and 335 b, each have curved surfaces 373that interact locally to help register or align adjacent camera channelstogether. In this example, to further reduce the potential forover-constraint, the channel to channel datum pairs are asymmetrical andpartially offset and are thus more likely to provide localized pointcontacts and less likely to cause over-constraint. It is noted that thecircular holes 530 provide access through a side wall of a lens housing430, to enable application of adhesive or RTV used in mounting an outerlens element 437 to the housing.

During assembly and alignment of the camera channels, it is alsoimportant to properly position the lens elements within the housings.For example, the compressor lens (437) can be provided with datumfeatures along edges 432 that interact with a set of mating datumfeatures on the inside of the lens housing 430. The datum featuresprovided on the lens elements are intended to limit perpendicularity andconcentricity error and are designed to be reasonably machined or groundfeatures, or to be mounted onto the ground glass lens bevel surfaceswith an adhesive. As shown in FIG. 16, these features can include flatdatum surfaces 650 fabricated on the bottom face of the compressor lensat the corner, outside the FOV. Other datums, including adjacent flatedges 660, that can be ground into the round lens element before it wasshaped into a pentagon with truncated or beveled edges 370, can be matedwith flat surface datums 670 on the lens mount. As glass is typicallymounted to metals with adhesives, these flat edge datums can provide aguiding alignment without risk of over constraint. The outer lenselement 437 can also be provided with an anti-clocking datum feature680, likely bonded to one of the beveled surfaces.

Because of these datum features, the chamfered or beveled finishes onthe pentagonal lens surfaces can be fabricated with significantly lessprecision, thus helping to reduce lens cost. Even though FIG. 9 depictsa pentagon-shaped compressor lens or outer lens element 437 and housing430, these mechanical approaches to reduce mis-alignment errors can alsobe applied to hexagonally shaped lenses, or lens elements having otherpolygonal shapes.

Alternately, or in addition, the lens housings 430 can be equipped withone or more tab or post like structures (not shown) that can protrudeout from a lens housing out of the nominal conical space or volume andinteract with similar protruding structures of an adjacent lens housing,or alternately interact with indented structures of an adjacent lenshousing. For example, these protruding structures can be provided in thegeneral vicinity of the sensor 270 and sensor package 265, and beseveral millimeters in length, and have datum features to helpkinematically control a DOF, such as a radial position from the devicecenter, or atilt or rotation of one camera channel to an adjacent camerachannel. A camera assembly can have two such protruding structures,arranged symmetrically or asymmetrically about the lens housing and oneoriented orthogonally to the other, to control a different degree offreedom. Alternately, or in addition, the camera channels can have lenshousings 430 that include one or more protruding tab, or post structureslocated within the seams 400. For example, such datum structures (notshown) can be provided within the seams, at the polygonal outer face ofa camera channel, at a vertex 60, and can protrude out of the nominalconical space or volume and into the seam 400 between the two adjacentchannels. Depending on the device design and intended applications, theprotruding tab or structures, whether located in the outer seams 400, ormore deeply imbedded, such as near the image sensors, can be either befabricated from a compliant or a stiff material, or a combinationthereof. As another alternative, one or more tabs of a given lenshousing 430 need not protrude outside the nominal conical volume orfrustum, but clamps that bridge from one tab to a tab of an adjacentlens housing can provide the interface or control to limit a degree offreedom.

FIG. 19 depicts an alternate version of the interface of the camerachannels to the mounting structure near the device center to that shownin FIG. 9 and FIG. 17. In particular, FIG. 19 depicts portions of twoadjacent camera channels, an upper primary channel 610 in cross-sectionand a secondary channel 615 in partial perspective. In this case, atripod or channel centering hub 330 has a socket 545 that provides aconcave surface that contacts a nominally matching convex surface of acentral hub 550. The central hub 550 is part of a mount mechanicalassembly nominally positioned with its center at the device center, andit can be attached to a support post 750. A cable 560, having a ball 565at an end, can be used to pull or tension the socket 545 against thecentral hub 550. The primary channel, and each of the secondarychannels, can be held together by similar tensioned cables that descendinto the support post, where they are fastened and locked. Alternately,the primary channel 610 can be held in place with a tightened bolt (notshown). As compared to the prior approaches of FIG. 9 and FIG. 17, thisapproach has exchanged the plurality of balls and sockets for aninverted configuration with a plurality of sockets 545 (one per camerachannel) contacting one main ball or hub 550. The tensioned cablesreplace the prior approaches that used retention pins, or latches, orsprings. The approach of FIG. 19 can enable numerous camera channels tobe simultaneously and robustly pulled into alignment against the ballhub 550, and about the device center. The ball hub 550 can be machinedfrom a precision ball bearing.

It is noted that in the assembly approach depicted in FIG. 9 and FIG.17, with a primary channel 610, and secondary channels 615, andcentering hubs 330 interfacing with a central ball socket array 346,that the available space in the center of the device, into which powerand communications cables, cooling lines, and support mechanics must fitcan be tight.

As shown in FIG. 13B, an improved camera 920, having a track length 980between the front lens center and the image plane 950, can be positionedat an offset distance 925 from the image plane 950 to the low parallaxvolume 992. As one approach to improve the device center congestion, thecamera 920, its housing 430 (not shown), and the overall improved device300, can be designed to provide an axial center offset distance 915along the optical axis 985 between the low parallax volume 992 and thedevice center 910, much as previously discussed with respect to FIG. 5C.Designing in an offset distance 915 (e.g., 1-4 mm) can provide extraspace for power, communications, or cooling connections, cables, and formechanical supports for the sensor package 952, or for other structuralmembers. In the example of the camera system 920 depicted in FIG. 13B,the improved low-parallax multi-camera panoramic capture device (300)can have a dodecahedral format, and then the device center 910 is thecenter of the nominal dodecahedral polygonal structure. This offsetdistance must be determined during the process of designing the cameras920 and overall device 300, as it interacts with the optimization of thelens near the edges of the FOV. Thus this optimization can depend on, orinteracts with, the seam width, the distortion correction, the controlof front color, the optimization for reduced parallax for peripheralrays (edge ray 972) or image rays 975 generally, or for the extent andsub-structure of the LP volume 992, the sizing of the lens elements(particularly for the compressor lens group 955), or the allowance foran extended FOV 215.

As another option to provide more access for cabling, supports, andthermal management hardware, the ball and socket approach of FIG. 17 canbe replaced with an internal frame (FIG. 20) with polygonal faces, withaccess holes to a hollow center. For an improved multi-camera panoramicimage capture device 300 constructed in a dodecahedral pattern, theinternal frame would also be dodecahedral with pentagonal faces and itwould be oriented with the internal pentagonal faces nominally alignedwith the external pentagonal geometry. An internal frame can be machinedseparately and assembled from 2 or more pieces, or it can be made as asingle piece structure by casting or 3D printing. Although, thefabrication of a single piece frame could be more complex, the resultingstructure can be more rigid and robust, and support tighter mechanicaltolerances. For example, a dodecahedral frame with a hollow center couldbe cast in stainless steel, and then selectively post-casting machinedto provide precision datum features. This internal frame can then beprovided with flexures or adjustors on all or most of the pentagonalfaces, to provide kinematic type adjustments and to reduce or avoid overconstraint during device assembly and use. As before, the availableadjustors on these internal faces can be different for the secondarychannels as compared to a primary channel. Alternately, an internalframe can be at least in part made with a more compliant material, suchas brass or Invar. As the central volume of this internal frame can beat least partially hollow, space can then be provided for the electricalcabling, thermal management hardware, and other support structures.

FIG. 20 provides an example of such an internal frame 800, with numerouspentagonal faces 810 arranged in a dodecahedral pattern with a hollowcenter. An internal frame 800 can be designed as a mount mechanicalassembly for an 11-camera system, with a support post attaching in the12th position (similar to FIG. 9 and FIG. 15). A polygonal internalframe, or half or partial internal frame can also be used in a partialor hemispheric system, where the camera assemblies, including imagessensors are mounted to the frame. Alternately, a hemispherical system(e.g., see FIG. 21) with an internal frame 800 can use a central hollowspace (e.g., a nexus) to enable image light to cross through in reachingimage sensors on the far side, including by transiting intervening relaylens systems (725). As shown in FIG. 20, a pentagonal face (810A) canhave three adjustors 820, such as set screws or flexures, orientednominally 120° apart, that can interact with features on the camerahousing and thus be used to help align a given camera channel. Aspreviously, the mounting and adjustments for secondary channels can havea different design or configuration than those for a primary channel. Asanother alternative (not shown), one or more pentagonal faces 810A,810B, or 810C can be provided with one or more adjustors that can beused to nudge the respective camera channel against a precision v-groovestructure (also not shown). These-v-groove structures can be fabricatedinto, or protruding from, an inside edge of a pentagonal vertex 60 of apentagonal face. The internal frame approach can be used with otherpolygonal device structures, such as that for an icosahedron.

FIG. 11 depicts an advantageous hardware configuration in the region ofthe sensor 270, cover glass 272, and the accompanying sensor package 265(which can include electronics, cooling and mounting). The cover glass272 can seal or protect the sensor from the environment. The cover glass272 can also provide UV or IR cut filtering by means of a thin filminterference or dichroic coating, or that function can be provided on aseparate window, external filter 295. The UV or IR cut filter reducesthe level of the non-visible light incident, that is accompanying theimage light 415, to the sensor 270. Alternately, or in additionally, UVand IR cut filtering can be provided with a coating applied to a lenselement, including to the outer surface of outer lens element 437 (FIG.9). The cover glass or filter 295 can also be a UV light absorbing glassand provide UV filtering by a combination of absorption and coatingreflectance.

It is recognized that in providing an improved multi-camera panoramicimage capture device 300 with outer lens elements 437 having innerbeveled edges 370 (FIGS. 10 and 16), that the centering tolerances canbe worse as compared to typical lens elements fabricated withconventional cylindrical edges. Such decentering can in turn effect thecentering of the FOV captured by the entire camera lens system 320. Asone approach for compensation of such errors, the positioning of theeffective image centroid or center pixel can be determined eitheroptically or electronically (FIG. 11). As another approach, an improvedlens housing 430 can be designed to provide a compensating lateraladjustment for one or more internal lens elements. For example, means toadjust the positioning or tilt of an intermediate inner lens element,such as for a lens element located between the stop and the imagesensor, can be provided. The adjustment means can include or usemicrometers, pins, shims, flexures, or springs. Z-axis compensators toadjust focus or magnification differences can also be provided withineach camera channel by similar mechanisms. The mechanisms to enablecompensation can be built in, or internal to the device or a camera, orexternal, or a combination thereof.

Although the opto-mechanics of FIGS. 8-10 and FIGS. 16-20 can reducealignment errors for the sensors 270, the camera housings 430, and oneor more cameras 320, these design improvements, and other comparableones, may not provide sufficient accuracy for all configurations orapplications of an improved multi-camera capture device 300. As another,or complimentary approach, an optical fiducial system can be provided.In particular, as shown in FIG. 11, a light source 460 can direct light462 for an optical fiducial into a window or filter 295, within a camera320 of an improved multi-camera panoramic image capture device 300. Theoptical fiducial light 462 can be coupled into an edge of the filter 295and propagates by total internal reflection (TIR) to an output coupler465, by which it is directed towards the sensor 270, where it providesone or more illuminated spots or areas that function as opticalfiducials 475. The optical fiducial light 462 can be low power infrared(IR) light, such as at 785 or 835 nm, and the output coupler 465 can beone or more lenslets, a prismatic feature, or a diffraction grating. Theilluminated areas that function as optical fiducials 475 can be focusedspots that are only a few sensor pixels wide. The optical fiducial light462 can be provided by a light source 460 that is mounted in amechanically stable position relative to the image sensor 270. Fiduciallight 462 that remains within the optical substrate of the filter 295,and is light guided towards the opposite edge, can be absorbed by anabsorber 470, which can for example be provided with a black paintcoating.

FIG. 11 depicts one advantageous approach for providing an opticalfiducial 475 that enables FOV tuning for a camera 320, and thus helpslimit parallax errors and image overlap or underlap for adjacent camerasin an improved panoramic multi-camera capture device 300. But ingeneral, an optical fiducial can be provided by a light source locatedproximate to the sensor. As another example, a light source could bemounted on or near the sensor plane, and direct light outwards towardsor through the cover glass, so that it reflects off of an opticalelement, and back towards the image sensor. The light source would bepositioned outside the used active area of the sensor, as would areflective coating. The reflective coating can be a localized metallicor dielectric coating positioned outside the clear aperture that is usedby transiting image light. As discussed previously, the light source canthen provide at least one illuminating spot of light onto active sensorpixels located outside, but proximate to, the active area used by imagelight, thereby providing an optical fiducial 475. This concept can alsobe extended, and optical fiducials can be attached to, or interact with,other components within the system, including lens elements or lenshousing structures. As such, the relative motion of particular lenselements or sub-groups thereof, could be monitored to inform the imagetracking, cropping, or correction efforts. Additionally, if one or morelens sub-groups or compensators can be actively driven, such as withmotors, the resulting data can be useful to inform those corrections.

As shown in FIG. 12, a camera 320 for an improved multi-camera panoramicimage capture device 300 can also be equipped with a mask or an internalbaffle 455, which can be located between an outer lens element 437, orcompressor lens, and the subsequent inner lens elements 440. Assuggested by FIG. 9 and FIG. 16, the beveled edges 370 of the outer lenselement 137 can be fabricated through a curved outer lens surface, at aset distance from, and parallel to, the nominal edge chief rays 170 thatare incident along a straight pentagonal outer edge. The beveled edges370 can have a curved bevel shapes or profiles. However, it is easierand cheaper to fabricate the outer lens elements 437, and mount themwithin lens housings 430, if the ground edges (370) have straight bevelsor chamfers. This however means that the volume of optical glass along astraight edge of a pentagonal (or hexagonal) outer lens element 437 thatcan accept light, that then may become stray light, or image light thatcomplicates image mosaicing or tiling, can vary along the beveled lensedges. The baffle 455, provided in FIG. 12, can provide a sharppolygonal edged aperture (e.g., pentagonal or hexagonal) following theshape of the outer lens element 437 and the core FOV 205, and also ablackened surface for blocking and absorbing light outside the intendedFOV. Alternatively, the light absorbing baffle 455 can be painted orcoated on an internal lens element surface. Thus, the baffle 455 or maskcan also define the edges of the transiting image light and thus cast anedge shadow 495 onto the image sensor 270 (see FIG. 11).

The optical fiducial light 462 and the shadow 495 cast by the baffle 455can be used both individually and in combination to provide an improvedmulti-camera capture device 300, with reduced parallax errors andreduced image overlap or underlap between cameras. As shown in FIG. 11,incident image light 415 can provide an illuminated image area 450 thatis incident to an image sensor that has an active area length and width.For example, a camera 320 that collects uniform image light within anominally pentagon-shaped core FOV 205 can then provide apentagon-shaped illuminated area 450 on the sensor 270. This image orilluminated area 450 can underfill a width of the sensor 270, to agreater or lesser extent, depending on the shapes of the illuminatedarea and the sensor. To be a useful datum, all or most of the projectededge shadow 495 can define boundaries of an illuminated area of activepixels within an underfilled sensor area. In some examples, the edgeshadow 495 fills most (e.g., ≥97%) of the active area width. The baffle455 is acting like a field stop. In cameras 320 without a baffle, theouter lens element can act like the field stop and define the polygonalFOV.

Within the illuminated areas corresponding to the edge shadow 495, thereis a smaller illuminated area 450 corresponding to the core FOV 205, andbetween them, there can be an intermediate extended FOV 215. Theextended FOV 215 can be large enough to almost match the size of theedge shadow 495. The difference can be optimized depending on theexpected mechanical alignment tolerances for the lens assembly and forpositioning the baffle 455, and the optical sharpness or transitionwidth of the projected shadow. Underfill of the sensor active area widthallows the core FOV 205, an image centroid, the extended FOV 215, andedge shadow 495 to shift position on the sensor without being clipped.It can be beneficial during a calibration process to establish an imagecentroid 480 that can then be tracked versus time and exposureconditions. In particular, an initial distance or offset 485 can bedetermined during a calibration process and stored as a pixel count.Then, if the illuminated image area 450 shifts during or after assemblyof the camera 320 into a multi-camera capture device 300, whether due tomechanical or thermal reasons, a new location of an image centroid 480and the shape, size, and position of the core FOV 205 can be determinedand compared to prior saved calibration data.

However, in completing this process, a boundary of the illuminated area450 that corresponds to the desired image data must be determined. Theedge shadow 495 cast by the baffle 455, much as suggested in FIG. 7, canthen provide a useful series of fiducial edges or points 210 proximateto, but a bit larger than, the core FOV 205. The shape and position ofthe shadow or occlusion cast by the baffle 455 can be determined duringa bench test of a given camera 320, prior to assembly of that camera 320into a given multi-camera capture device 300. Similar calibration datacan be obtained after that assembly, and then again over time as thecamera and device are used, to track changes in the shadow positioningthat might occur due to environmental or mechanical initiated changes inthe internal mounting or positioning of the lens elements (e.g., 437,440) or the baffle 455.

In greater detail, the FOV edge defining baffle 455 and the opticalfiducial 475 (see FIGS. 11 and 12), can be used in combination tomonitor or longitudinally track the position of the image centroid 480and image area edges, to aid image mosaicing or tiling. In particular,for the purposes of aligning a camera lens 320 for both center androtation with respect to an image sensor 270, a method of projecting anoptical fiducial or casting an optical occlusion onto the image sensor270, can be used either individually or in combination.

Essentially the baffle 455 or mask casts a multi-edged shadow onto thesensor 270, and the shape of the baffle will be sampled from around theperiphery of the projected image or illuminated area 450 captured by thesensor 270 in software or hardware. The shadow edge can be relativelysharp, but can still create a gradient fall off, potentially spanning atleast several pixels, between the illuminated area 450 and the darksurround of the projected shadow. The data for the sampled edges orfiducials from around the edges of the occluded area and the shadowgradients can then be used to derive an image centroid 480 of the lensprojection. The baffle 455 or occluding shadow 495 can also have anadded feature to indicate and derive rotation if needed. A calibrationstep can be initially used to derive a relationship of the shadow 495 tothe lens center and rotation. Additionally, the dimensions and shape ofthe mask can be accurately measured before installation and compared tothe size and shape of the projected shadow 495. The derived relationshipof the size, shape, and centroid of the projected shadow 495 can accountfor an expected or measured difference in the shape and size of the edgeshadow 495 compared to the shape and size of the baffle. Other factorsthat can affect the edge shadow 495, such as mask tilt, and an impact oflens distortion in a fractional field portion (e.g., 0.85-1.01)corresponding to the outer portion of a core FOV 205 and an extended FOV215, can also be accounted for.

Additionally, as previously described, an optical fiducial 475 can beprojected using IR or visible light onto an unused portion of the sensor270. The pattern formed by the fiducial can then be used to derive thelocation of the fiducial and be calibrated to the image centroid 480,and the lens center and rotation. A calibration step is initially usedto derive the relationship of the optical fiducial 475 to the lenscenter and rotation and correlated to the distortion characteristicscalculated in the calibration process. Additionally, a series ofcalibration images of the fiducial shadow 495 provided by baffle 455 canbe used to locate more distant features (e.g., the corners), and thus toconfirm or determine the planarity of the lens to sensor mechanicalalignment. Sensor alignment or planarity can be modified using theadjustments previously described in FIG. 8. The combined fiducialmethod, of using both an optical fiducial 475 and a projected fiducialshadow 495 has the advantage of being more robust in diverse lightingconditions where the edge of the occlusion cast method may beinconsistent or hard to detect.

The method for calibrating with optical fiducials (or with electronicfiducials) and shadow occlusions can be used for a given camera toaccurately align the modeled lens distortion derived from a calibrationstep to the calibrated core FOV 205. The resulting correction data canbe stored in matrices or other forms in an on-board look up table (LUT)on a local board that is part of the sensor package, for each camera andsensor. When an image is captured, this correction data can be used tocrop a larger image, corresponding to an extended FOV 215, during aninitial or interim capture step, down to an image corresponding to areal current core FOV 205. Likewise, other available data, such as forexposure and color corrections, that can be stored locally, can also beapplied in real time to the cropped, core FOV sized image, before it istransmitted to a system computer for further image processing, includingimage seaming or tiling to assemble a panoramic image.

More broadly, in general, during either a real time or post processingstep of modifying image data captured by one or more cameras 320 of animproved multi-camera panoramic capture device 300, a method forcalibrating optical fiducials and shadow occlusions can be used for agiven camera to accurately align the modeled lens distortion derivedfrom a calibration step to the captured image. Then an image can beaccurately undistorted as needed, and also corrected to account forother camera intrinsics (e.g., lens focal length, or sensor parameters).This method can be replicated for the multiple cameras 320 of amulti-camera capture device 300 to enable accurate mosaicing or tilingof multiple images together. By matching knowledge of the cameraintrinsics accurately to the captured images, the quality of themosaicing across the boundaries or seams between images captured byadjacent cameras will increase the combined image quality and reduce theerrors from the image being misaligned to the calibrated intrinsicsinitially calculated. As such, the speed of the mosaicing or tilingincreases because little or no time can be spent on compute intensive(e.g., optical flow) image feature-based methods to correct formismatches in alignment, distortion, and lens intrinsics.

FIG. 11 also shows that outer portions of the active area or pixel areasof a rectangular image sensor 270 can be covered or shielded by masks455. In the case that the sensor 270 is much larger than the imageilluminated area 450, use of masks 455 prevents stray light fromotherwise being incident to the underlying dark areas. As a result,these pixel values will essentially be “zero” at all times, and theiroutput can be ignored or dumped, which can speed image processing andcompression times. Also, it is possible to replace or substitute thelight source 460 and optical fiducial 475 with an offset value 485 foran electronic fiducial calculated as a difference in a calibratedlocation of the centroid 480 to a sensor edge or mask edge. Use of anelectronic fiducial can be simpler and cheaper to implement, as comparedto the optical fiducial 475, but depending on the mechanical design ofthe camera housing 430, or a use case for the improved multi-cameracapture device 300, it may or may not be as mechanically or functionallyrobust.

Variations in brightness from scene to scene, or camera to camera, canaffect the image content incident within an illuminated area 450corresponding to a core FOV 205. In particular, image quality and imagemosaicing or tiling can vary with content; such as dark scenes to onecamera, and a bright scene to an adjacent camera, affecting imagemosaicing and exposure levels. These variations not only make it hard todetect image or shadow edges, but also to make it difficult to determineimage centroid values to use during image seaming. Furthermore, theseexposure variations can make mosaiced images appear tiled along oracross the edges. While either an electronic or optical fiducial can beused to help determine image centroids and image edges, an opticalfiducial can also be used to enable electronic exposure controlcorrection. Additional exposure correction can be provided by matchingthe center or average values of adjacent cameras and scenes, to help themosaiced images blend or match better.

The masks 455 depicted as part of a sensor portion of a camera 320 canbe eliminated or reduced if the sensor 270 has a configuration closer tothat of the core FOV 205 or the shape of the image illuminated area 450.Replacing a rectangular sensor with an alternate one that is square, ornearly so, makes more effective use of the available space. Inparticular, much smaller portions of the conical volume that a cameralens assembly can nominally be designed to fit within are lost insupporting unused sensor area. The illuminated area 450 can land on asensor with a more appropriate active area, and thus increase theeffective resolution (in pixels at the sensor, or in pixels/degreerelative to the core FOV 205). However, use a of square image sensor canstill provide space or pixels to support use of an optical fiducial 475.Alternately, the rectangular image sensor can be replaced with a sensorhaving an active area optimized for the camera shape, such as to have ahexagonal or pentagonal shape. Having a shape optimized sensor canfurther ease the optical design, thus making it easier to fit the cameraassembly within the nominal conical volume or frustum, and to optimizethe “NP point” location and size. Use of a shaped optimized sensor canprovide greater freedom for the optical design optimization trade-offsor balance for factors including the relative distance between the imagesensor and the NP point, image quality, pupil aberrations, imagedistortion, and ray constraints for reduced parallax. As a result, theoptical resolution (e.g., from the lens) and angular resolution (fromthe sensor) can be further optimized, and exceed 100 px/deg. However,the design can still have the core FOV 205 underfill the optimized shapesensor, to allow room for image capture of an extended FOV, and to avoidloss of potential image content due to image shifts and rotations. Theunderfill, if large enough, can still also allow space to provide anoptical fiducial 475.

It is noted that for many image sensors 270, the image light is incidentdirectly onto the sensor pixels. However, many commercially availableimage sensors are further equipped with an integrated lenslet array,overlaying, and aligned to, the pixel array. A given lenslet directs itsportion of incident image light to the corresponding underlying pixel.This approach can reduce optical crosstalk between pixels, as theincident image light is more likely to be photo-electrically convertedinto an electrical signal within the incident pixel than may have beentrue otherwise. As the pixels have become progressively smaller, thisapproach has become increasingly common, as a means to improve ormaintain image resolution (MTF). Such sensors with integrated lensletarrays can be used in the cameras 320 of the improved multi-cameracapture devices 300.

Alternately, an overlaying lenslet array can provide an array oflenslets, where any given lenslet directs light onto a plurality ofimage sensor pixels. In this case, direct image resolution can be lost,but with a benefit of gaining a light field capability to provide eitherstereo image capture or increased depth of focus image capture. Thistype of light field capability can be designed into a panoramicmulti-camera capture device 300, but much of the benefit can then belost to FOV overlap (FIG. 3). In such systems, the optical designs alsocan have the entrance pupils at the front or outer lens elements, whichcan lead to information loss. However, introducing light fieldcapabilities into an improved panoramic multi-camera capture device 300that has a plurality of adjacent abutting cameras 320 with reducedparallax errors, whether from improved kinematic design, improvedopto-mechanical design at the seams 400, or the use of electronic oroptical fiducials or shadow edge masks, can improve the performance andvalue of captured light field data for such systems. Particularly, insuch a system, cameras 320 with light field imaging can capture aportion of the available plenoptic light to provide images with animproved depth of focus and object perspective, without either theinformation loss or complicated image fusing that results from parallaxerrors.

As noted previously, the width of the seams 400 can be a criticalparameter in the design of an improved multi-camera capture device 300,as the seam width affects the parallax error, size of blind regions 165,the size of any compensating image overlap (FIG. 3), and the imageprocessing and mosaicing or tiling time. For example, in cases where anexpensive multi-camera capture device 300 is used in controlledenvironments, narrower seams with small tolerances could be allowed ortolerated. However, a local collision or stress of expensive brittlematerials (e.g., glass) should be avoided, as chipping and other damagecan result. There are of course other external damage risks, includingthose from external materials coming in contact with the outer lenselements 437. Thus, as a multi-camera capture device 300 can be anexpensive unit, opto-mechanical designs that provide risk mitigation, orprotection of the cameras or the device, can be beneficial.

As one remedy, a seam 400 can be filled with a compliant material, suchas an RTV or silicone type adhesive. As another remedy, the opticaldesign of the camera lenses can include an outer lens element 437 thatis designed to be a polymer or plastic material, such as Zeonex, that isless brittle than is glass. The combined use of a polymer outer lenselement 437 and seams 400 filled with compliant materials can furtherreduce risks.

However, in optical designs, glass usually provides better and morepredictable performance than do polymer materials. As one approach, theouter lens element 437 may be designed with a common, more robust andless expensive optical glass, such as BK7. The outer lens elements,whether glass or polymer, can also be overcoated with scratch resistantAR coatings, and oleophobic or hydrophobic coatings. A compliant seamfilling adhesive should likewise resist penetration or contamination bywater, oils, or other common materials.

It can be advantageous to design the camera lenses 320 with lenselements 435 that include an outer lens element 437, or compressor lens,is more of a meniscus type lens element than previously suggested (FIG.2A). As an example, as shown in FIG. 13A, the compressor lens elementhas been split into a compressor lens group, including a firstcompressor lens element (437) and compressor lens elements 438 that arecombined as a doublet. The outer lens element 437 can then more freelybe designed to have a partially meniscus type shape, while stillproviding both some beam or image light compression or re-directiontowards a wide angle lens group that include a fourth lens element 442,and the chief light collection to reduce parallax. If the outer lenselement is more meniscus like, or generally has a long focal length, itcan be flatter, and protrude less above the lens housing 430. Thus, itcan also be less likely to accept light rays 410 at extreme angles fromobject space 405, but from outside the nominal field of view 425, thatcan successfully reach an image sensor 270 as visible or detectableghost light. In such a lens design, a second compressor lens element438, or doublet, provides further optical power, and re-directs thetransiting image light inwards towards an inner fourth lens element 442.

Additionally, the camera lenses 320 can be opto-mechanically designed sothat the outer lens element 437 can be a field replaceable unit (FRU).One way to aid this approach is to design that lens element to have amore meniscus like shape. The optical mechanical design can be designedto enable an outer lens element 437 to be mounted so that it can also bereadily removed and replaced if it is damaged. For example, the outerlens element can be directly mounted into the camera lens housing 430,using datum features such as those shown in FIG. 16, but then beremovable using one or more extraction tools. A desolvable adhesive(e.g., glyptal) or hot glue, with a transition temperature aboveterrestrial extremes, can also be used. A replacement outer lens elementcan then be mounted in its place. Alternately, the outer lens elementcan also be in an assembly, mounted to a sub-housing portion that can beremoved from the main camera housing 430. In either case, providing alens design in which the outer lens element has a partial meniscus orlong focal length design then reduces the sensitivity of the design,relative to image quality and image location on the sensor, to aninaccurate alignment of a substitute outer lens element. Potentiallythen, a cast or molded optical element, whether made with a polymer(e.g., acrylic) or a glass (e.g., B270), can also be used. Camerare-calibration using the previously discussed optical or electronicfiducials, or the shadow 495 of a baffle 455, further reduces the risksassociated with outer lens replacement.

As another alternative for an improved multi-camera capture device 300,an entire camera 320 can also be designed to be modular, or a potentialFRU. In such a case, the camera 320, including the housing 430, and theaccompanying lens elements including the outermost compressor lenselement 437, can be removed and replaced as a unit. For example, thepressure from the channel loading support 630 can be released orreduced, and the ball pivot of a camera channel can then be releasedfrom a ball socket 345, and a replacement camera channel can be insertedin its place. Thereafter, the pressure from the channel loading support630 can be restored. As the relative positions of the camera channelscan shift slightly during this re-assembly process, a process of FOVcentering or calibration with fiducials may have to be repeatedthereafter. Depending on the complexity of the design, FRU replacementof modular camera lens assemblies may occur in the field, in a servicecenter, or in the factory, but with relative ease and alacrity.

An improved multi-camera capture device 300, and the cameras 320therein, can also be protected by a dome or a shell (not shown) withnominally concentric inner and outer spherical surfaces through whichthe device can image. The addition of an outer dome can be used toenclose the nearly spherical device of FIG. 15, or for a nearlyhemispheric device like that of FIG. 21, or a device having an alternategeometry or total FOV, within an interior volume. The dome can consistof a pair of mating hemispheric or nearly hemispheric domes thatinterface at a joint, or be a single nearly hemispheric shell (e.g., forFIG. 21). The transparent dome or shell material can be glass, plasticor polymer, a hybrid or reinforced polymer material, or a robust opticalmaterial like ceramic, sapphire, or Alon. The optically clear dome orshell can help keep out environmental contaminants, and likewise ifdamaged, function as a FRU and be replaced. It can be easier to replacea FRU dome than an entire camera 320 or a FRU type outer lens element orouter lens element assembly. The dome or shell can also be enhanced withAR, oleophobic, or hydrophobic coatings on the outer surface, and ARcoatings on the inner surface. Although the use of a dome or shell canreduce the need or burden of also using a carrying case or shippingcontainer, such enclosures can still be useful. Alternately, or inaddition, the dome or shell can be faceted, and provide a series ofcontiguous adjacent lens elements, which can function as the outer lenselements for the associated adjacent camera systems 320. This approachcan have the potential advantages of reducing the widths of both theintervening seams 160 (e.g., seam widths ≤0.5 mm) and their associatedblind regions 165 and enabling the device center 196 to be coincidentwith the low parallax volumes 188.

As yet other alternatives, shown in FIGS. 14A and 14B, an improvedmulti-camera capture device 300, and the cameras 320 and their housings430 therein, can be designed to enable providing protective fins 510that protrude from the seams 400, or protective posts 520 that protrudefrom the vertices (60) or corners where a plurality of adjacent camerasmeet. These fins 510 or posts 520 would protrude into object space 405,but can be designed to be outside the captured FOV, by protruding lessthan the extent of a blind region 165, and they could prevent directcontact by some objects from the outside environment. These fins orposts can also be designed to be rigid enough, with the right aspectratios of height to thickness, to reduce their risk of buckling.Alternately, the fins or posts can be designed with a somewhat compliantmaterial so as to rebound from a contacting force, or a combination,such as compliant fins and rigid posts can be used in tandem. The postsand fins can be supported by compliant mounting that is imbedded withthe seams 400, so that these protective features can revert towardstheir native designed shape without damaging the cameras 320 ormulti-camera capture device 300 during an impact event. Use of darkenedfins that protrude, by perhaps a few or tens of millimeters, from theseams 400, can also provide a benefit of reducing acceptance of lightrays at extreme incidence angles relative to a camera lens, that canbecome ghost light that successfully reaches an image sensor 270. Use offins or posts will only protect the cameras 320 and the multi-cameracapture device 300 from a moderate force or impact, and thus for somedesigns and applications, there could be sufficient value. These finsFIG. 14A or posts of FIG. 14 B can also be sacrificial elements ordetachable FRUs themselves. Once they have done their job, or ifdamaged, they can then be substituted by plugging in replacements (e.g.,perhaps into receptacle sockets designed specifically for that purpose).

A prior example was given in which a proposed width of a seam 400 wasrather narrow. But to clarify, relative to the design and performance ofa multi-camera capture device 300, for chief rays 170 transitingadjacent cameras 320, and for a given object distance, it can beunnecessary to hold the beam to beam separation to complete zero.Rather, a goal can be to have at most a pixel of missing information (ora few pixels, depending on the application) at the object distance ofinterest. Depending on the allowed seam width and pixel loss, thecameras 320, and multi-camera capture device 300, can be more or lessrobust, or more or less tolerant of both fabrication tolerances andprotective design approaches, including those discussed previously.

In particular, some fabrication tolerances of the cameras 320 (e.g., seeFIG. 9), housings 430, outer lens elements 437, and seams 400 can betolerated or accommodated. In part, the use of the optical or electronicfiducials, or a shadow cast by a baffle, enables greater tolerance orflexibility in collecting low parallax error images from adjacentcameras 320 with little FOV overlap, as these corrective approaches canenable a “centered” image to be found. However, a multi-camera capturedevice 300 can also be tolerant of some image loss. For example, for a 5foot object distance, and a device whose cameras support an aggregate 32Megapixel resolution (an 8k output equirectangular image), a 1 pixelwidth seam 400 can equate to a 1.2 mm sized gap. If an improvedmulti-camera capture device 300 is designed to hold the mechanical seamsto 3 mm after construction, then the cameras can be designed for a 1.5mm gap while allowing some FOV overlap (□□). The device can have amechanical design that with tolerances, allows a matching pointing errorof at most □□. After the cameras are assembled, the actual rays that arecollected parallel to the edge surface that are shared between thecamera lenses may shift, but will always be within the pointing error.Thus, a gap or seam 400 can be constrained to be at most 3 mm wide. Forsome camera designs, and markets or applications, wider seams can betolerated, in either absolute size (e.g., 4.5 mm seam widths) or in lostimage content (e.g., 2-20 pixels per seam). Lost pixels can becompensated for by increasing the FOV overlap or extended FOV 215,between adjacent cameras, to capture overlapped content but at a cost ofsome parallax errors and some increase in the image processing burden.Although, for modest amounts of extended FOV (e.g., ≤5%), the residualparallax errors (e.g., FIG. 8B) can still be modest. The smaller theseams 400, and the better the knowledge of the position of an imagecentroid 480, and an image size and shape, then the larger the core FOV205 can be on the sensor, and the less lens capacity and sensor area canthen be devoted to providing a yet larger extended FOV 215.

The seams 400 can be nominally identically wide at the interfacesbetween abutting adjacent cameras for all cameras 320 in an improvedmulti-camera capture device 300. Careful camera and camera housingdesign, and camera to camera mounting (FIGS. 9 and 10) can help this tooccur. But nonetheless, the seam widths can vary, either upon deviceassembly, or during dynamic environmental conditions. For example, onone side of a first camera, a seam width between that camera and anadjacent one can be only 0.75 pixels wide, while simultaneously a seamwidth between the first camera and another adjacent camera can be 3.25pixels wide. The seam widths can also vary non-uniformly. The resultingimages to the sensors can be shifted or rotated relative toexpectations. Thus, such variations can increase the parallax errors forthe images captured by these adjacent cameras, and complicate imagemosaicing or tiling. However, with the use of electronic or opticalfiducials (475), or shadows cast by internal baffles 455, an imagecentroid and image edges (FIGS. 11 and 12) can be monitored for eachcamera to enable a quick reference to the nominally calibrated orexpected conditions. This reference or correction data can also becompared from one camera to another, with a goal to define effectiveimage centroids and image edges for each camera that effectively bestreduces parallax errors for all cameras, either individually, or inaggregate (e.g., average). This information can then be used duringimage processing to quickly and robustly determine image edges andenable efficient image mosaicing or tiling.

As discussed previously, with respect to FIG. 15, a useful configurationfor a multi-camera capture device can be to design and fabricate agenerally spherical system, having a plurality of camera lensesdistributed in a dodecahedron or truncated icosahedron arrangement.However, an issue that can occur with an improved multi-camera capturedevice 300 of the types of FIG. 15, is that with the plurality of camerachannels and respective image sensors confined within a nominallyspherical shape, there is little room to include other components orcapabilities. Thus, for some applications, devices with a generallyhemispherical configuration, with potential room underneath for otherhardware, can be valuable. However, because the outer lens elements andcameras are typically polygonal in shape, a hemispherical device canhave a jagged or irregular circumference. Also, in such systems, one ormore of the cameras can be designed with folds (e.g., using mirrors orprisms) so that the optical paths extend through the bottom irregularcircumferential surface. This construction can provide more room for useof modular sensor units that can be swapped in and out.

However, for a “hemispheric” version of a truncated icosahedron, thereare 6 camera channels with pentagonal faces, and 10 with hexagonalfaces, and it can be difficult to provide space for the opto-mechanicsto fold so many optical paths. FIG. 21 depicts an alternative version ofan improved multi-camera capture device 700 in which image lightcollected by the respective camera objective lens systems 720 isdirected along a nominally straight optical path, through the nominalimage plane, and then through an image relay optical system (725) to amore distant image sensor located in a sensor housing 730. The originalimage plane provided by an objective or camera lens system isessentially a real intermediate image plane within the larger opticalsystem. It can be re-imaged with a magnification (e.g., 1:1 or 2.75:1)to a subsequent image plane (not shown) where an image sensor islocated. Thus, advantageously, the image sensors can be larger, andprovide a higher pixel count, and the relay lens systems 725 canre-image the image provided by the objective lenses (720) at anappropriate magnification to nominally fill the more distant sensor witha projected image. FIG. 21 depicts an upper “hemispheric” portion of atruncated icosahedron, with a semi-internal view that reveals portionsof two camera objective lens systems 720, each with an associated relaysystem 725 that extends outside the nominal conical space or volume.Image light from the respective cameras 720 crosses through a centralvolume or nexus 710 on its way through the respective relay lens systems725. The relay lens systems 725 can include field lenses (not shown)after the image plane of the camera lenses, but before the nexus, tohelp contain the space needed for transiting image light. The optics ofthe relay lens systems 725 can also be coherently designed with thecameras 720, so as to correct or compensate for aberrations thereof.

To provide a hollow central volume or nexus 710 for the imaging lightrays from the adjacent camera channels to cross one another and transitthe respective relay lens systems 725, an internal frame 800 (e.g., seeFIG. 20) with polygonal faces having access holes to allow the imagelight to transit the hollow center. As previously suggested, an internalpolygonal frame can have flexures or adjustors provided on all or mostof the polygonal faces, to provide kinematic or quasi-kinematicadjustments and to reduce or avoid over constraint during deviceassembly and use. However, in this case the hollow space or centralvolume of this internal frame is primarily provided so as to allow imagelight rays to transit the central nexus 710, and secondarily can alsoproviding room for electrical cabling and thermal management hardware.Alternately, mirrors (not shown) can also be used in the relay lenspaths to redirect the image light so that the overall opto-mechanicalstructure is more compact. The improved multi-camera capture device 700also includes a support structure 740, a support post 750, and cabling760 for supplying power and extracting signal (image) data. The supportstructure 740 can provide more substantial support of the camerahousings 730 than is illustrated in FIG. 21.

FIG. 15 depicts an electronics system diagram for an improvedmulti-camera capture device 300. In this example, a dodecahedron typedevice has 11 cameras 320, and an electro-mechanical interface in thetwelfth camera position. Image data can be collected from each of the 11cameras, and directed through an interface input-output module, througha cable or bundle of cables, to a portable computer that can provideimage processing, including live image cropping and mosaicing or tiling,as well as camera and device control. The output image data can bedirected to an image display, a VR headset, or to further computers,located locally or remotely. Electrical power and cooling can also beprovided as needed.

Also, as suggested previously, the performance of a multi-camera capturedevice, relative to both the opto-mechanics and image quality, can beaffected by both internal or external environmental factors. Each of theimage sensors 270, and the entirety of the sensor package 265, with thedata interface and power support electronics can be localized heatsources. To reduce the thermal impact on the camera lenses, and theimages they provide, the mechanical design for an improved multi-cameracapture device 300 can isolate the sensors 270 thermally from the lensopto-mechanics. To further help reduce thermal gradients between thesensors and their electronics, and the optics, micro-heat pipes orPeltier devices can be used to cool the sensors and re-direct the heat.The heat can be removed from the overall device by either active orpassive cooling provided through the electro-mechanical interface in thetwelfth camera position, shown in FIG. 15. This cooling can be providedby convection or conduction (including liquid cooling) or a combinationthereof.

As also suggested previously, outside ambient or environmental factorscan also affect performance of a multi-camera capture device. Thesefactors can include the effects of the illuminating ambient light, orthermal extremes or changes in the environment. For example, as sunlight is typically highly directional, a scenario with outdoor imagecapture can result in the cameras on one side of the device beingbrightly illuminated, while the other cameras seeing plenopticillumination from the scene are in shadows. In such instances, thecaptured images can show dramatic exposure variations, which can then bemodified by exposure correction, which can be provided locally (see FIG.15). In an improved multi-camera capture device 300, light from anoptical fiducial 475 can also be used for exposure correction ofcaptured images. Light or pixel signals from a portion of the peripheralimage region, between the edges of the core FOV 205 and the extended FOV215, can also be used for exposure correction, prior to the image beingcropped down to the size of the real current core FOV 205. It is alsonoted that as extended FOV 215 of a first camera can overlap at least inpart with an extended FOV 215 of an adjacent camera, that light leveland color comparisons can be made for content or signals that aresimultaneously captured by both cameras. The signals or pixel data fromthese overlapping regions can be used to determine exposure variationsbetween the two cameras, by having a common reference point (e.g., amatched feature point—using SIFT, SURF or a similar algorithm forfinding common feature points in the overlap region).

It is noted that the peripheral image or exposure data can also beretained for use in later post image processing. Additionally, exposurecorrection can also be enabled by imbedding optical detectors in theseams 400, or vertices, between outer lens elements 437. These abruptexposure differences can also cause spatial and temporal differences inthe thermal loading of some image sensors 270, as compared to others,within a multi-camera capture device 300. The previously discussedsensor cooling, whether enabled by heat pipes, heat sinks, liquidcooling, or other means, can be designed to account for suchdifferences. The performance can be validated by finite element analysis(FEA).

Alternately, one or more camera systems can be protected by theattachment of a shield or mask to cover the polygonal shape, from seamto seam, and vertex to vertex, of the outer lens element thereof. Suchshields can be provided to cover a single camera lens system, ormultiple systems. These shields can be shaped to generally conform tothe outer surface shape of the outer lens elements, and they can be usedto prevent saturated exposure or overexposure from bright directionallight (e.g., solar), or to prevent contamination from a localizeddirectional environmental source. While these caps are nominallydetachable, for some user applications, they may remain in use for aprolonged time period. Overly bright exposure, from the sun or otherlight sources can also be controlled with an image sensor havingelectronic shuttering or drains, or a physical shutter or anelectro-optical neutral density, photochromic, or electrochromic filter,that can, for example be designed into a camera 320, within for examplethe grouping of inner lens elements 440. Signals to initiate or controlelectronic or secondary shuttering can be obtained from the image sensoror from other internal or external optical detectors. As anotherrobustness improvement, one or more camera channels can use dichroiccolor filter arrays integrated into the image sensor package instead ofthe standard dye-based CFAs.

Environmental influences can also cause a multi-camera capture device tobe heated or cooled asymmetrically. The previously discussed nominallykinematic mounting or linkage of adjacent camera housings 430 (see FIGS.9-10 and FIGS. 17-18) for an improved multi-camera capture device 300can help reduce this impact, by trying to deflect or average mechanicalstresses and limit mechanical displacements. However, it can beadditionally beneficial to provide channels or materials to communicateor shift an asymmetrical thermal load to be shared more evenly betweenor by cameras 320 and their housings 430. With respect to FIG. 9, thiscan mean that the spaces around the lens housing 430 and the channelcentering hub 330, such as the inner volume 390, can be at leastpartially filled with compliant but high thermal contact, thermallyconductive materials (e.g., Sil-Pad (from Henkel Corporation) orCoolTherm (Lord Corporation, Cary N.C.)) to help spatially average anasymmetrical thermal load or difference. Alternately, or additionally,thermal conductive straps or tapes, such as an adhesive tape in the 88xxseries from 3M (St. Paul, Minn.) can be used. However, at the same time,some of the effect of thermal changes, relative to the imagingperformance of the camera lenses 320, can be mitigated by both judiciousselection of optical glasses and athermal mounting of the opticalelements within the lens housing 430. Taken in combination, an effectivedesign approach can be to enable thermal communication or crosstalkbetween lenses 320 and their housings 430 to environmental influences,but to simultaneously isolate the lenses and housings from the sensors270 and their electronics.

An improved camera 320 for use in an improved multi-camera image capturedevice 300 can also use a tunable lens element to correct for thermallyor mechanically induced focus changes (e.g., defocus). This tunable lenscan preferentially be located amongst the inner lens elements 440, andcan be a liquid crystal or elastic polymer type device, such as anelectrically actuated focus tunable lens from Optotune (Dietikon, SW).

The emphasis has been on developing improved cameras 320, that have apolygonal shaped outer lens element and capture and image light from apolygonal FOV, for use in an improved multi-camera image capture device300. Multiple such adjacent cameras can be used in a nominally sphericaldevice or hemispherical device. However, devices 300 can be developedthat have a yet smaller number of cameras and that cover a yet smallertotal FOV. For example, systems with only four or six low-parallax,adjacent polygonal cameras can be suitable for some market applications.Additionally, a single camera having a polygonal shaped outer lenselement that captures image light with reduced parallax or perspectiveerror, from a nominally matching polygonal shaped FOV can be use inisolation, such as for security or surveillance applications. Forexample, a single camera, optomechanically designed to fit within theform of ⅛^(th) of an octahedron can be mounted in a ceiling corner of aroom, and capture image content of the room environment with little orno blind regions. Likewise, the emphasis has been on the development ofimproved camera lens systems 320 for which parallax errors can bereduced within at least a core FOV 205, while modest extended FOVs 215(e.g., ≤5% extra) and image capture overlap with adjacent cameras canalso be provided. Similarly, it is noted that the present approach canbe extended to support possible applications having yet largeroverlapping FOVs (e.g., 10-25% extra, or ≈4-10° extra FOV fordodecahedral systems with a nominal Core FOV of 37.45°) of image capturebetween adjacent cameras, and simultaneous parallax error or perspectiveerror control within at least a core FOV. The camera designs can beextended even further, to provide yet larger overlapping FOVs (e.g.,10-20° extra), but without benefit of reduce parallax for angles muchbeyond the designed core FOV.

As noted previously, the present approach can be used to design lowparallax lens systems for a variety of applications. A method of devicedevelopment can be outlined, for example, as follows:

1. Define for a customer or an application space, a polygonal deviceconfiguration, a resolution, an expected extent of the seams and blindregions, and a minimum object distance. The minimum object distance isthe closest distance at which instant image stitching or image combiningof low parallax overlapped images can be applied if image processing isacceptable. At this distance, estimate a maximum total gap or seam widthfor which the image losses or differences are generally imperceptible toa human viewer. For example, for an 8k equirectangular projected image,an expected image loss of 1/40th % of the output image (along theequator) equates to seams that are 2 pixels wide each. An alternateminimum object distance, for which even modest image processing is notacceptable, and which is further away, can be defined.

2. Design the imaging lenses, while controlling parallax and frontcolor. The design can be further modified to provide an extra orextended FOV in which the parallax can be controlled. This extra FOV canprovide margin for the rainbow tinting of the residual front color to belocated outside of the Core FOV with some margin.

3. Design the lens and device opto-mechanics. Determine an expected seamwidth and the expected thickness and wedge variations between and withinthe seams, from fabrication and assembly tolerance variations. Determinean expected extended FOV overlap to cover the expected mechanical andoptical seam widths to be as high or greater than the maximum expectedopto-mechanical wedge and thickness variations so that parallel chiefrays between adjacent lens systems can be selected without underlap(fields of view that diverge).

4. Update the lens design to provide field of view overlap along theseams between the outer compressor lens elements, to exceed both themaximum expected mechanical variations and residual front color, whilecontrolling parallax therein.

In conclusion, aspects of this disclosure provide improved panoramicmulti-camera capture devices having low parallax cameras in which theopto-mechanics synergistically enable image capture by the cameras. Anintervening mechanical seam between two adjacent cameras can have a realwidth and thus impact the ray splitting or the extent of blind regionsand number of lost pixels or the value of having an extended FOV. Thepresent application provides several means, structures, andconfigurations to provide robust and controlled alignment of multipleadjacent cameras within a panoramic device, in part so that the seamscan be reliably reduced in width.

In order to push chief rays to the edge of a polygonal surface,aberrations of the entrance pupil, and particularly spherical aberrationand axial chromatic aberration of the pupil, should be optimized orreduced. For context, the entrance pupil and exit pupil, which areprojected images of the aperture stop in object space and image spacerespectively, are usually academic curiosities, representing the virtualimages of the aperture stop that a person can see when looking into alens.

In a typical optical system, to provide good image quality, aberrationsmatter at an image plane, with the typical goal to have small net sumsof the individual surface contributions, even though values atindividual internal surfaces, whether positive or negative, can bemagnitudes larger. Whereas, aberrations at the aperture stop often donot matter much, other than positioning the stop correctly and definingthe lens system f-number while minimizing clipping or vignetting of theray bundles versus field. It is noted that if an object was located atthe aperture stop, pupil aberrations would affect the apparent imagequality of the image of that object as seen by the person viewing theentrance or exit pupil, although these pupil aberrations are notnecessarily representative of the image quality at the image plane.

In low-parallax lenses, on the other hand, pupil aberrations, andparticularly entrance pupil aberrations, matter. First, to begin toproperly design the lens to control parallax, the entrance pupil must bepositioned behind the image plane. Secondly, the aiming of peripheralchief rays from the outer compressor lens element, and towards the lowparallax volume, must be controlled. As noted previously, optimizingspherical aberration of the entrance pupil can be an effective way tolimit parallax errors. In that context, modest over-corrected orunder-corrected optimizing spherical aberration of the entrance pupilcan occur, meaning that the non-paraxial chief ray NP points can lead orfollow the paraxial NP point respectively. Additionally, axial chromaticaberration of the entrance pupil creates color differences that canaffect optimization of spherical aberration of the entrance pupil. Frontcolor, which can be considered to be an artifact of this axial chromaticaberration, and the axial chromatic aberration itself, can be reduced byjudicious choice and use of high and low dispersion optical materialswithin the lens design. Although optimization of the sphericalaberration of the entrance pupil or chief ray pointing provides finecontrol for the non-paraxial, chief ray NP point position and parallaxreduction, distortion from the compressor lens group also has anincreasing influence, versus increasing field, on the projected chiefray pointing towards a location behind the lens and towards the lowparallax volume. The magnitude and characteristics of the distortion,which also defines the chief ray height on the outer surface of theouter compressor lens element, can be significantly determined by theuse of aspheric surfaces within the compressor lens group.

Although this discussion has emphasized the design of improvedmulti-camera image capture devices 300 for use in broadband visiblelight, or human perceivable applications, these devices can also bedesigned for narrowband visible applications (modified using spectralfilters, ultraviolet (UV), or infrared (IR) optical imagingapplications). Polarizers or polarizer arrays can also be used.Additionally, although the imaging cameras 320 have been described asusing all refractive designs, the optical designs can also bereflective, or catadioptric and use a combination of refractive andreflective optical elements.

What is claimed is:
 1. An imaging device comprising: a first imaginglens comprising: a first outer lens element having a polygonal shapecomprising a plurality of first sides, and a first datum feature on afirst side of the plurality of sides; a second imaging lens comprising:a second outer lens element having a polygonal shape comprising aplurality of second sides, and a second datum feature on a second sideof the plurality of sides, wherein the first datum feature and thesecond datum feature maintain a spacing between the first side and thesecond side.
 2. The imaging device of claim 1, wherein the first datumfeature contacts the second datum feature.
 3. The imaging device ofclaim 1, wherein the first datum feature comprises one or more firstconvexly curved protrusions and the second datum feature comprises oneor more second convexly curved protrusions.
 4. The imaging device ofclaim 1, further comprising: a first housing coupled to the firstimaging lens; a second housing coupled to the second imaging lens; and amounting structure located proximate to the center of the imagingdevice, wherein the first housing and the second housing are coupled tothe mounting structure.
 5. The imaging device of claim 4, wherein: themounting structure comprises a convex outer surface, the first housingcomprises a first concave surface receiving a first portion of theconvex outer surface of the mounting structure, and the second housingcomprises a second concave surface receiving a second portion of theconvex outer surface of the mounting structure.
 6. The imaging device ofclaim 4, wherein the mounting structure comprises a plurality of concaveportions formed in an outer surface, the first housing comprises a firstconvex outer surface received in a first concave portion of theplurality of concave portions, and the second housing comprises a secondconvex outer surface received in a second concave portion of theplurality of concave portions.
 7. The imaging device of claim 4, furthercomprising a first locking feature for retaining the first housingrelative to the mounting structure and a second locking feature forretaining the second housing relative to the mounting structure.
 8. Theimaging device of claim 7, wherein at least one of the first lockingfeature or the second locking feature includes at least one of a spring,a latch, or a cable.
 9. The imaging device of claim 4, wherein themounting structure comprises an internal frame having a polygonal shapethat nominally matches a polygonal shape of the imaging device, withpolygonal faces nominally matching the polygonal shapes of the firstouter lens element and the second outer lens element.
 10. The imagingdevice of claim 1, wherein: first projections of first non-paraxialchief rays included in incident light that enters the first outer lenselement converge in a first low parallax volume; second projections ofsecond non-paraxial chief rays included in incident light that entersthe second outer lens element converge in a second low parallax volume;and the first and second low parallax volumes at least partiallyoverlap.
 11. The imaging device of claim 10, wherein a firstnon-paraxial point associated with the first imaging lens and a secondnon-paraxial point associated with the second imaging lens are nominallycoincident with a center of the imaging device.
 12. The imaging deviceof claim 1, further comprising: an optical fiducial for at least one oftracking a first image centroid at a first image plane of the firstimaging lens, tracking a second image centroid at a second image planeof the second imaging lens, or exposure correction.
 13. The imagingdevice of claim 1, further comprising: an optically transparent domedefining an interior volume, the first imaging lens and the secondimaging lens being disposed in the interior volume.
 14. The imagingdevice of claim 1, wherein: the first imaging lens images a first fieldof view comprising a first nominal field of view and a first extendedfield of view larger than the first nominal field of view; the secondimaging lens images a second field of view comprising a second nominalfield of view and a second extended field of view larger than the secondnominal field of view; the first extended field of view and the secondextended field of view overlap at the seam between the first imaginglens and the second imaging lens; and a residual parallax error withinthe overlap of the first extended field of view and the second extendedfield of view is two pixels or less.
 15. A low-parallax imaging devicecomprising: a first camera configured to image a first polygonal fieldof view, the first camera comprising a first housing and a first outerlens having a plurality of first sides defining a first polygonalperiphery, wherein the first camera converges a first projection ofincident non-paraxial chief rays to a first non-paraxial point within afirst low-parallax volume; a second camera configured to image a secondpolygonal field of view, the second camera comprising a second housingand a second outer lens having a plurality of second sides defining asecond polygonal periphery, a second side of the plurality of secondsides contacting a first side of the plurality of first sides at a seambetween the first camera and the second camera, wherein the secondcamera converges a second projection of incident non-paraxial chief raysto a second non-paraxial point within a second low-parallax volume; anda mounting structure proximate to a center of the imaging device, themounting structure configured to position the first housing relative tothe second housing such that the first low-parallax volume at leastpartially overlaps the second low-parallax volume.
 16. The low-parallaximaging device of claim 15, wherein the mounting structure comprises afirst surface for contacting the first housing and a second surface forcontacting the second housing.
 17. The low-parallax imaging device ofclaim 15, wherein the mounting structure defines a hollow center. 18.The low-parallax imaging device of claim 15, wherein the first cameraincludes a first datum feature on a first side of the plurality of firstsides that contacts a second datum feature on a second side of theplurality of second sides.
 19. The low-parallax imaging device of claim15, wherein the mounting structure comprises a channel centering hubwith sockets and the first camera housing and the second camera housinghave a ball feature that mate with the sockets.
 20. The low-parallaximaging device of claim 15, wherein the mounting structure comprises aball shaped central hub with socket features and the each of the firstand second camera housing have a concave socket feature thatrespectively mates with the ball shaped central hub.